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form  of  computerized  mathematical  models  to  assist  in  predicting  the 
quantitative  impact  of  training  resource  decisions.  The  planning  process 
will  be  enhanced  by  providing  decision  makers  with  the  capability  to 
economically  and  rapidly  consider  a wider  range  of  alternatives. 

Phase  I was  a study  and  definition  effort  resulting  in  a complete  func- 
tional description  of  the  NAVEDTRACOM;  a strategic  definition  of  the 
social,  political,  economic  and  technological  environments  pertinent  to 
Naval  education  and  training  system  in  the  1 980 ' s ; a list  of  existing 
and  potential  models  amenable  to  computerization  and  to  improving  the 
decision-making  process.  Phase  II  was  devoted  to  the  selection  and 
development  of  three  mathematical  models  from  the  Phase  I list  of  poten- 
tial candidates.  The  three  were  the  System  Capabilities/ Requirements 
and  Resources  (SCRR),  the  Educational  Technology  Evaluation  (ETE),  and 
the  Training  Process  Flow  (TPF)  models. 

During  Phase  III,  the  three  models  were  installed  at  an  operational  test 
bed  (Fleet  Training  Center,  Norfolk,  Virginia)  using  a time-sharing 
tele-processing  system.  Based  upon  real-world  data,  the  management 
application  of  the  models  was  evaluated  by  a Test  and  Evaluation  (T&E) 
team  comprised  of  representatives  of  the  Chief  of  Navy  Education  and 
Training  (CNET)  and  CNET  functional  commands,  led  by  the  Naval  Personnel 
R&D  Center  (Washington  Branch). 

In  summary,  the  T&E  team  concluded  that  the  DOTS  models  established, 
beyond  reasonable  doubt,  the  feasibility  of  applying  modeling  techniques 
to  Navy  training  problems.  The  team  recommended  that  the  next  step 
should  be  a field  test  proving  the  validity  of  the  DOTS'  models  at  the 
functional  command  level.  It  was  also  recommended  that  the  name  of  the 
ETE  model  be  changed,  and  as  a result  it  is  now  entitled  the  Individual- 
ized Training  System  Simulation  (ITSS)  model. 
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FOREWORD 


The  Design  of  Training  Systems  (DOTS)  project  objectives  are  in  consonance 
with  the  requirements  of  Advanced  Development  Objective  ZPN07  (formerly 
ADO  43-03X),  Education  and  Training  Developmen.  ZPN07  includes  a number 
of  projects  concerned  with  demonstrating  and  evaluating  the  technical,  opera- 
tional and  financial  feasibility  of  applying  advanced  technological  applications 
to  improving  the  training  process. 

The  Bureau  of  Naval  Personnel  initiated  the  original  ADO  in  1966  to  make  Naval 
training  more  responsive  to  the  changing  times.  As  one  project  under  this  effort, 
DOTS  was  designed  to  improve  the  process  of  managing  training  resources  through 
application  of  the  techniques  of  system  analysis  and  system  simulation  as  accom- 
plished through  mathematical  modeling.  The  end  objective  is  a family  of  computer- 
ized mathematical  models  enabling  training  management  to  more  rapidly  predict 
the  impact  of  changes  in  training  resource  availability  or  requirements. 

The  majority  of  education  and  training  was  reorganized  in  1971  under  one  com- 
mand, Chief  of  Naval  Education  and  Training  (CNET).  Because  of  this  change, 

DOTS  responsibility  was  transferred  to  CNET  in  March  of  1972,  more  specifically 
to  the  Training  Analysis  and  Evaluation  Group  (TAEG),  Orlando,  Florida.  The 
new  CNET  organization  greatly  increased  the  potential  benefits  to  be  gained 
from  the  increased  application  of  new  management  techniques  and,  therefore, 
from  the  DOTS'  R&D  effort.  Accomplishment  of  DOTS  began  in  February  of  1973 
with  the  majority  of  tasking  being  assigned  to  the  International  Business 
Machines  Corporation,  Federal  Systems  Division,  Cape  Kennedy  Facility,  located 
at  Cape  Canaveral,  Florida. 

In  conducting  the  Phase  I study  and  definition  effort,  the  TAEG/IBM  technical 
team  conducted  multi-level  interviews  at  some  eighty  activities  or  training 
related  groups  within  the  Naval  Education  and  Training  Command  (NAVEDTRACOM) . 

The  willing  and  competent  participation  of  all  personnel  contacted  is  gratefully 
acknowledged.  Special  recognition  is  due  the  Fleet  Training  Center,  Norfolk, 
Virginia,  which  was  selected  as  a test  bed  for  the  DOTS'  models.  All  levels  of 
the  command  were  exceptionally  cooperative  and  willing  to  contribute  significant 
time  and  interest  to  the  implementation  of  the  models  and  their  subsequent  test 
and  evaluation  during  Phase  III. 

The  SCRR,  ETE,  and  TPF  models  were  developed  by  Mr.  R.  Yanko,  Mr.  H.  Bellamy, 
and  Mr.  K.  Branch,  respectively.  Systems  Programming  support  was  provided  to  the 
modelers  by  Mr.  J.  Chapman  and  Mr.  J.  Staley.  Mr.  C.  Edison  developed  and  im- 
plemented the  Phase  III  training  program.  All  team  members  participated  in  the 
Phase  III  documentation  effort,  with  Mrs.  D.  Gardner,  Mrs.  L.  Girard,  Mrs.  C. 
Reilly  and  Mrs.  E.  Taylor  providing  editorial  and  secretarial  services. 

Mr.  R.  Hallman  was  Project  Manager. 

The  Training  Analysis  and  Evaluation  Group,  Dr.  A.  Smode,  Director,  project 
team  members  Mr.  H.  Okraski , Mr.  T.  McNaney  and  Mr.  W.  Lindahl,  complemented 
the  contracted  effort  by  providing  guidance  and  establishing  organizational 
interfaces. 
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SECTION  I 
INTRODUCTION 


BACKGROUND 

The  Department  of  Defense  (DoD)  Is  faced  with  maintaining  a strong  national 
defense  posture  despite  declining  allocations  and  the  impact  of  world 
inflation.  The  increasing  cost  of  complex  weapons  systems  and  support  man- 
power is  significantly  increasing  the  challenge  of  meeting  national  defense 
objectives. 

Approximately  145S1  of  the  DoD  annual  budget  is  allocated  to  some  form  of 
education  or  training.  This  represents  about  25%  of  DoD's  total  manpower 
budget.  Obviously,  a major  strategic  thrust  of  the  military  services  is  to 
reduce  education  and  training  costs  to  a minimum,  while  maintaining  the  re- 
quired level  of  effectiveness. 

The  majority  of  Naval  training  is  under  the  command  of  the  Chief  of  Naval 
Education  and  Training  (CNET).  The  CNET  is  responsible  for  providing  the 
training  support  required  to  maintain  fleet  readiness  within  the  constraints 
of  available  funding.  The  Design  of  Training  Systems  (DOTS)  project  Is  intended 
to  assist  the  Naval  Education  and  Training  Command  (NAVEDTRACOM)  in  achieving 
this  goal . 

The  overall  DOTS'  objective  is  to  provide  NAVEDTRACOM  management  with  additional 
tools  in  the  form  of  computerized  mathematical  models  to  asssist  In  predicting 
the  quantitative  impact  of  training  resource  decisions.  The  planning  process 
will  be  enhanced  by  providing  decision  makers  with  the  capability  to  economically 
and  rapidly  consider  a wider  range  of  alternatives.  The  project  stresses  a 
step-wise  progression  from  system  analysis,  through  development  of  computer- 
based  test  models  of  selected  sub-elements  within  the  NAVEDTRACOM,  to  recom- 
mendations for  making  the  models  operational. 

Phase  I of  the  DOTS  project  was  completed  in  December  1973.  It  included: 

(1)  a comprehensive  functional  analysis  of  the  NAVEDTRACOM,  (2)  development 
of  a set  of  strategic  assumptions  describing  the  environmental  elements  expected 
to  affect  the  NAVEDTRACOM  In  the  1980's,  (3;  development  of  recommendations 
leading  to  an  idealized  training  system  in  terms  of  the  projected  needs  of  the 
1980's,  and  (4)  creation  of  a candidate  list  of  potential  computer-based  math- 
ematical models  to  permit  selection  of  a maximum  range  test  configuration.  If 
further  information  is  desired,  consult  the  DOTS  Phase  I Final  Report2. 

Phase  II  was  completed  In  October  1974.  It  Included:  (1)  selection  of  three 
test  models,  from  the  candidate  list  developed  during  Phase  I,  representing 


^Defense  Space  Daily,  17  January  1974,  Page  92. 

^Design  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 
December  1973. 
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a range  of  subsets  of  the  NAVEDTRACOM  resource  planning  and  management  process, 
(2)  development  and  logical  validation  of  the  selected  test  models,  (3)  iden- 
tification of  those  areas  within  the  NAVEDTRACOM  which  could  not  be  modeled 
nor  well  defined,  suitable  modeling  techniques  do  not  exist,  or  data  for  model 
input  or  testing  is  not  readily  available,  (4)  development  of  a hypothetical 
cost  avoidance  analysis  projecting  the  impact  of  modeling  tools  operating 
within  a modified  NAVEDTRACOM  management  system,  (5)  development  of  the 
design  for  verification  studies  to  be  carried  out  during  Phase  III.  The 
three  selected  test  models  were  the  System  Capabilities/Requirements  and 
Resources  (SCRR),  the  Educational  Technology  Evaluation  (ETE),  and  the  Train- 
ing Process  Flow  (TPF)  models.  Additional  information  may  be  obtained  from 
the  DOTS  Phase  II  Final  Report3. 

Phase  III  was  completed  in  October  1975.  Phase  III  included  development  of  the 
unique  software  required  to  implement  the  DOTS'  models  in  the  Norfolk  test  bed, 
the  orientation  and  training  of  operating  personnel  and  members  of  the  Govern- 
ment Test  and  Evaluation  (T&E ) team,  support  of  the  T&E,  and  final  user  and 
maintenance  documentation. 

The  majority  of  the  technical  data  concerning  the  three  DOTS  models  is  contained 
in  the  previously  mentioned  documentation.  A brief  description  is  presented  in 
SECTION  IV,  PHASE  III  PRODUCTS.  The  emphasis  of  this  Phase  III  report  will  be 
on  describing  the  T&E  task,  summarizing  its  results  and  suggesting  recommenda- 
tions for  future  DOTS  direction. 


3 

Design  of  Training  Systems,  Phase  II  Final  Report.  TAEG  Report  No.  12-2, 
December  1974. 
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PROBLEM  STATEMENT 

The  direct  result  of  the  DOTS  effort,  as  implied  by  its  objective,  is  envisioned 
as  a family  of  computerized  mathematical  models  providing  a new  capability  to 
predict  the  quantitative  impact  of  training  resource  decisions.  The  indirect 
result  will  be  the  use  of  model  outputs  by  training  managers  to  significantly 
increase  the  number  of  alternatives  considered  in  arriving  at  resource  decisions. 
The  overall  resultant  will  be  more  efficient  use  of  training  resources. 

The  three  DOTS  models  were  selected  as  representing  a cross-section  of  the 
significant  planning  decisions  addressable  by  computerized  mathematical  models. 
Using  the  selected  models  as  test  vehicles,  the  T&E  of  Phase  III  was  concerned 
with  their  evaluation  within  two  problem  areas: 

• NAVEDTRACOM  Decision  Process 

The  first  problem  area  was  concerned  with  the  control  and  planning 
of  requirements  and  resources.  The  Department  of  Defense  (DoD)  Pro- 
gramming, Planning  and  Budgeting  System  (PPBS)  is  the  primary 
NAVEDTRACOM  driver.  As  the  result  of  various  PPBS  actions,  the 
NAVEDTRACOM  resource  planning  and  control  process  is  directly 
impacted  with  appropriate  management  actions  being  initiated  at 
multiple  levels.  These  "appropriate  management  actions"  are 
based  on  information  flowing  from  both  lower  and  higher  levels  of 
organization. 

The  T&E  team  was  tasked  with  determining  the  degree  of  capability 
of  the  DOTS  models  for  assisting  the  decision  process  within  the 
framework  of  this  system.  In  addition,  recommendations  as  to 
level  and  techniques  of  application  were  to  be  developed. 

• NAVEDTRACOM  Operational  Environment 

The  second  problem  area  was  concerned  with  the  practical  Implications 
of  implementing  computerized  mathematical  models  in  terms  of  finan- 
cial feasibility  and  the  operational  Impact  on  support  personnel. 

The  T&E  was  tasked  with  evaluating  the  DOTS  models  based  on  observa- 
tions as  they  were  operated  in  the  Norfolk  FLETRACEN  test  bed. 

Within  the  context  of  these  two  problem  areas,  the  Phase  III  T&E  provided  guid- 
ance to  the  future  direction  of  DOTS  as  well  as  to  an  assessment  of  the  three 
models.  The  T&E  results  are  summarized  in  SECTION  III,  TEST  AND  EVALUATION 
RESULTS. 
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SECTION  II 

TEST  AND  EVALUATION  APPROACH 


OBJECTIVE 

The  T&E  team  objective  was  a very  significant  one  in  relation  to  the  overall 
DOTS'  R&D  effort  as  represented  by  Phases  I,  II  and  III.  Through  the  team's 
assessment  of  the  three  DOTS  models,  conclusions  and  recommendations  were 
drawn  pertinent  to  the  two  problem  areas  defined  under  SECTION  I's  PROBLEM 
STATEMENT  subsection.  The  T&E  results  also  provided  a basis  for  making  more 
general  recommendations  pertinent  to  the  application  of  managerial  science, 
operations  research,  and  mathematical  modeling  techniques  to  the  NAVEDTRACOM 
planning  and  decision  process. 

In  support  of  the  overall  DOTS'  objective,  Phase  Ill's  major  tasks  were: 

§ To  develop  and  document  conclusions  pertinent  to  the  DOTS'  hypothesis 
which  states  that  the  application  of  managerial  science,  operations 
research,  and  mathematical  modeling  techniques  could  significantly 
enhance  the  planning  and  decision  process  within  the  NAVEDTRACOM 
resulting  in  a significant  gain  in  efficiency. 

• To  develop  recommendations  giving  direction  to  future  DOTS'  R&D 
efforts  as  well  as  possible  operational  implementation  of  DOTS 
type  models. 

Using  the  outputs  of  Phase  II,  validated  test  models  and  a selected  test  bed. 
Phase  III  activities  of  the  IBM  technical  team  required  to  achieve  the  above 
were: 

• To  design  and  develop  the  unique  software  required  to  support 
implementation  of  the  DOTS'  test  models  at  the  test  bed,  Norfolk 
Fleet  Training  Center  (FLETRACEN). 

• To  accomplish  orientation  and  training  of  Fleet  Training  Center 
personnel  and  members  of  the  Government  Test  and  Evaluation  Team. 

• To  develop  sufficient  technical  documentation  to  permit  continued 
development  or  modification  of  the  DOTS'  models  beyond  Phase  III. 

The  activities  of  the  Government  T&E  Team  are  implied  by  the  Problem  Statements 
previously  described.  The  Team's  objective  was  to  provide  an  evaluation  and/or 
judgmental  assessment  of  the  : (in  order  of  descending  priority) 

• Potential  of  the  DOTS  type  mathematical  model  and  the  systems 
approach  for  enhancing  the  management  of  resource  and  requirements 
within  the  total  NAVEDTRACOM. 

• Technical  validity  of  the  three  test  models. 

• Value  of  the  model's  capabilities,  assuming  reasonable  modifications 
to  the  present  management  system. 
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• Operational  practicality  and  financial  implications  of  the  DOTS  models 
operating  in  the  Norfolk  FLETRACEN. 

Based  on  the  T&E,  conclusions  and  recommendations  were  developed  and  are  summar 
ized  in  this  report.  The  potentially  operational  by-products  of  the  testing 
process  were  documented  in  two  separate  volumes:  a User's  Manual  and  a Program 

Maintenance  Manual.  These  are  described  in  Section  IV  of  this  document. 
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SUPPORTING  TASKS 

To  give  perspective  to  the  Phase  III  tasks,  this  subsection  will  present  an 
overview  of  the  DOTS'  tasks  accomplished  during  Phases  I and  II  but  pertinent 
to  the  Phase  III  T&E.  The  following  chart  will  assist  In  maintaining  a time 
reference. 


DOTS  Taat  & Evaluation  Supporting  Tasks 
Phaaaal.il  and  III 
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T&E,  Phases  I and  II  Supporting  Tasks  (Cont'd) 


The  T&E  related  tasks  were: 

• Select  Test  Models  (Phases  I and  II) 

The  details  of  the  test  model  selection  process  are  covered  in  the 
Phase  I Report4  and  will  not  be  repeated  here.  The  three  selected 
models,  Educational  Technology  Evaluation  (ETE),  Training  Process 
Flow  (TPF)  and  System  Capabilities/Requirements  and  Resources 
(SCRR),  met  two  selection  principles  of  permitting  an  assessment 
of  the  DOTS  hypothesis,  amenable  to  extrapolative  analysis,  and 
of  providing  a potential  operational  capability.  The  greatest 
weight  was  given  to  selecting  models  most  supportive  of  the 
DOTS  test  objectives. 

The  selected  test  models  are  intended  to  be  complementary.  Three 
models,  as  opposed  to  one,  were  selected  to  facilitate  DOTS  hypothe 
sis  evaluation  across  a broader  spectrum  of  operation,  as  well  as 
of  multi-level  application.  Although  none  of  the  three  test  models 
is  time  restricted,  each  has  been  designed  to  emphasize  a range  of 
operational,  near  horizon,  or  long-range  planning.  The  types  of 
requirements  addressed  by  the  test  models  were  common  to  all  levels 
of  the  NAVEDTRACOM  and,  therefore,  extrapolation  of  results  beyond 
the  actual  test  bed  was  feasible. 

• Develop  Test  Models  (Phase  II) 

The  three  test  models  were  developed  during  Phase  II  to  an  opera- 
tional test  installation  level  and  are  described  in  the  Phase  II 
Final  Report5.  The  designs  were  based  on  operational  needs  identi- 
fied during  Phases  I and  II. 

• Validate  Test  Models  (Phase  II) 

The  purpose  of  the  validation  task  was  to  objectively  demonstrate 
that  all  sub-elements  of  the  three  test  models  would  perform  the 
functions  they  were  designed  to  do.  In  short,  did  the  models 
logically  perform  as  intended? 

The  validation  was  performed  in  a laboratory  environment,  i.e., 
validation  runs  were  made  on  a computer  and  with  data  remote  from 
the  ultimate  test  bed.  Validation  should  not  be  confused  with 
verification.  Validation  proves  the  basic  design  logic  of  a model, 
verification  proves  the  accuracy  of  a model's  predictions  or  its 
ability  to  simulate  a process  or  environment. 


^Design  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 
December  1973,  Pages  VII-1  through  VI 1-33 . 

5Pesign  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2, 
Vols.  2 and  3,  December  1974. 
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T&E,  Phases  I and  II  Supporting  Tasks  (Cont'd) 


Phase  II's  validation  task  is  described  in  the  Phase  II  Final 
Report®.  The  three  test  models  were  successfully  validated. 

• Select  T&E  Test  Bed  (Phase  II) 

Based  on  the  original  DOTS'  plan.  Phase  III  verification  was  to  have 
taken  place  in  the  same  laboratory  environment  as  the  Phase  II 
validation.  To  increase  accuracy,  it  was  decided  to  actually  per- 
form this  task  in  an  operational  test  bed.  The  approach  offered 
a secondary  benefit;  if  the  models  did  demonstrate  utility,  the 
potential  for  more  immediate  practical  application  would  be 
enhanced. 

The  change  to  an  operational  test  bed  also  permitted  expansion 
of  Phase  Ill's  scope  to  encompass  potential  human  factors  pertinent 
to  operational  use  as  well  as  technical  verification  of  the  test 
model's  fidelity. 

Based  on  the  empirical  observations  of  Phase  I,  there  seemed  to  be 
little  doubt  that  mathematical  models  could  be  developed  providing 
valid  predictive  information  for  the  NAVEDTRACOM  planning  and 
decision  process.  Phase  II's  validation  of  the  three  test  models 
subsequently  offered  empirical  support  to  this  supposition.  The 
most  difficult  issue  had  to  do  with  the  operational  implications  of 
various  management  levels  actually  using  the  models  within  the  existing 
organizational,  procedural  and  motivational  system.  This  issue  spanned 
a wide  range  of  concerns  from  the  practical  data  processing  support 
required  to  the  support  of  DOTS  concepts  by  various  management  levels. 

Plans  for  Phase  Ill's  formal  T&E  were  changed  to  permit  evaluation 
in  an  operational  test  bed  rather  than  experimentally  in  a labora- 
tory environment.  By  making  use  of  an  existing  commercial  time- 
sharing service  and  installing  a teleprocessing  display  terminal 
and  printer  in  an  operational  test  bed,  the  formal  T&E  evaluation 
could  be  based  on  observation  in  an  operating  environment  closely 
resembling  a projected  real-world  situation. 

Based  on  the  above,  a task  was  added  to  Phase  II  to  select  a 
viable  operational  test  bed.  The  Norfolk  FLETRACEN  was  selected 
from  a number  considered  since  it  most  closely  matched  the  entire 
span  of  considerations  and  was  of  a size  permitting  installation 
of  the  DOTS'  models  within  the  scope  of  available  resources. 

More  specifically,  Norfolk: 

- Had  an  extensive  mix  of  "A"  and  "C"  schools. 

- Supported  a fairly  broad  combination  of  ratings. 


^Design  of  Training  Systems,  Phase  II  Final  Report.  TAEG  Report  No.  12-2, 
Vo 1 . 2,  December  1974,  Pages  11-30-60,  III-19-32  and  IV-22-32. 
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T&E,  Phases  I and  II  Supporting  Tasks  (Cont'd) 


- Was  subjected  to  unpredicted  requirements  for  new  courses, 
new  quotas,  etc. 

- Was  responsible  for  all  functions  in  the  training  development 
cycle  between  course  design  and  course  implementation. 

- Was  involved  in  the  implementation  or  planning  of  new 
instructional  techniques  such  as  Individualized  Learning. 

- Had  acceptance  of  scientific  management  techniques  at  the 
command  level  and  a desire  to  participate  in  the  DOTS'  T&E. 

- Was  general  enough  in  level  and  type  to  be  representative 
of  a significant  fraction  of  Navy  training. 

- Was  of  a size,  permitting  development  of  a data  base  within 
the  scope  of  the  DOTS'  effort. 

The  three  test  models  were  selected  based  on  their  potential  for 
evaluating  the  original  DOTS'  hypothesis  and  not  for  their  immediate 
value  to  the  Norfolk  FLETRACEN.  To  partially  compensate  Norfolk 
for  its  interest  and  effort  in  supporting  the  test  and  evaluation, 
some  limited  software  was  developed  to  permit  access  to  the  DOTS 
supplemental  data  base  required  to  drive  the  test  models. 

The  selection,  development,  and  logic  validation  of  the  test  models, 
and  the  operational  test  bed  selection,  were  accomplished  during 
Phase  II.  The  following  tasks  supportive  of  the  T&E  were  completed 
during  Phase  III: 

• Test  Bed  Implementation 

This  task  included  selection  of  a time-sharing  service  and 
supporting  terminal,  development  of  Fleet  Training  Center, 
Norfolk  (FLETRACEN,  NORVA),  unique  software  and  data  base, 
training  of  operational  personnel,  and  initial  operational 
support  of  model  use. 

• Support  Navy  Test  and  Evaluation 

The  Navy  T&E  team  was  tasked  with  the  actual  T&E.  IBM  was 
tasked  with  the  development  of  the  T&E  plan  and  a training 
program  for  the  T&E  team  members. 

• Support  Documentation 

IBM  developed  sufficient  documentation  during  Phase  III  to 
assure  continuity  of  the  project,  whether  the  Intent  was 
additional  R&D  or  projects  leading  to  operational  imple- 
mentation of  the  models.  These  documents  are  described 
in  SECTION  IV,  PHASE  III  PRODUCTS. 
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T4E,  Phases  I and  II  Supporting  Tasks  (Cont'd) 


• T&E  Results  Analysis 

The  Government  T&E  Team  devoted  considerable  effort  and 
time,  after  their  Norfolk  training  and  test,  to  results 
analysis  and  documentation.  Its  output  was  a Final 
Report  which  is  summarized  in  SECTION  III,  TEST  AND 
EVALUATION,  TEST  AND  EVALUATION  SUMMARY. 
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TEST  AND  EVALUATION 


The  Government  T&E  Team  was  formalized  In  April  of  1975  and  consisted  of: 


F.  DiGialleonardo  - Team  Leader  - NPRDC  - Washington 
LCDR  R.  J.  Blersner,  Code  N-214  - CNETS  - Pensacola 
Lt.  R.  N.  Brooks  - TRALANT  - Norfolk 
CDR  J.  D.  Davis,  Code  015  - CNTECHTRA  - Memphis 
LCDR  T.  L.  Ferrier,  Code  N-31  - TRAPAC  - San  Diego 
Mr.  J.  T.  Finnlgan,  Code  N-31  - TRAPAC  - San  Diego 
Mr.  E.  Scheye,  Code  N-336  - CNET  - Pensacola 
Mr.  D.  S.  Thomas,  N-215  - CNTECHTRA  - Memphis 


The  T&E  Team  was  supported  by  the 

Mr.  William  Lindahl  - TAEG 
Mr.  Thomas  McNaney  - TAEG 
Mr.  Harold  Bellamy  - IBM 
Mr.  Kenneth  Branch  - IBM 
Mr.  Charles  Edison  - IBM 
Mr.  J.  D.  Staley  - IBM 
Mr.  Ronald  Yanko  - IBM 


following  IBM  and  TAEG  personnel: 


- Orlando 

- Orlando 

- Cape  Canaveral 

- Cape  Canaveral 

- Cape  Canaveral 

- Cape  Canaveral 


In  April,  the  T&E  team  was  provided  an  IBM-developed  test  plan^  and  goals. 
However,  under  direction  of  the  team  leader,  a unique  plan  and  goals  were 
developed  prior  to  Initiation  of  the  formal  T&E.  Generally,  the  differences 
were  In  degree  of  emphasis  rather  than  In  areas  and  concepts  to  be  tested. 


Both  plans  were  based  on  the  results  of  a joint  Norfolk  FLETRACEN/TAEG/IBM 
problem  Identification  workshop  held  In  December  of  1974.  After  being  intro- 
duced to  the  problem  solving  capabilities  of  the  DOTS  models,  requirements  and 
problem  statements  were  solicited  from  FLETRACEN  management  personnel.  The 
solicitation  was  not  restricted  to  only  those  problems  addressed  by  the  models. 
Responses  ranged  from  requests  for  specific  data  items  to  requirements  for 
sophisticated  analysis  of  interrelationships  between  personnel  characteristics 
and  academic  potential.  The  problems  Identified  were  concerned  with  such  areas  as 


• Impact  of  student  load  changes  on  instructor  requirements. 

• Relationship  of  contact  hours  to  the  number  of  instructors. 

• Impact  of  various  factors  on  the  attrition  rate. 

• Correlation  of  instructor  capabilities  to  potential  cross-training. 

• Relationship  of  personnel  cuts  to  student  load. 

• Relationship  of  convening  frequency  to  such  areas  as  utilization, 
capacity,  convenience,  attrltltlon,  etc. 


7pes1gn  of  Training  Systems,  Preliminary  User's  Test  Guide.  TAEG,  Section  III, 
31  March  1975. 
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Test  and  Evaluation  (Cont'd) 


The  schedule  of  formal  Phase  III  T&E  events  was  as  follows: 

• 4 June  through  6 June  1975:  Training  by  IBM  of  the  Norfolk 
FLETRACEN  staff. 

• 16  June  through  20  June  1975:  Training  by  IBM  of  the  T&E  team. 

• 23  June  through  27  June  1975:  T&E  team  model  analysis  at  the 
Norfolk  FLETRACEN. 

• 30  June  through  5 September  1975:  Visits  to  various  functional  com- 
mands by  the  T&E  team  leader,  accompanied  by  a member  of  the  Project 
Office,  and  documentation  of  the  final  results. 

The  approach  used  during  each  of  these  phases  was  as  follows: 

• Training  at  the  Norfolk  FLETRACEN  Staff 

Training  course  design  was  based  on  the  assumption  that  there  were 
three  types  of  personnel  concerned  with  model  application.  These 
were  the  executive  or  decision-making  level,  the  analyst  who 
could  provide  the  model  capability  interface  between  executive 
and  system,  and  the  operator  who  would  actually  perform  mechanical 
interface  operations  such  as  data  update  or  calling  forth  the 
models.  These  three  levels  did  not  include  the  responsibility  of 
technical  maintenance  of  the  models  since  it  was  assumed  that,  if 
the  models  were  subsequently  implemented  operationally,  maintenance 
would  take  place  at  a central  location  remote  from  the  operational 
user. 

The  executive  level  received  a limited  formal  introduction  to  the 
DOTS  system  with  the  objective  of  simply  creating  an  awareness 
of  the  DOTS  capability. 

The  analyst  level  received  two  days  of  intensive  orientation. 

The  emphasis  was  on  what  the  models  would  do  rather  than  how 
they  did  it.  Operator  training  concentrated  on  the  more  mechani- 
cal aspects  of  system  manipulation.  It  should  be  remembered  that 
the  purpose  of  training  the  Norfolk  personnel  was  to  permit  the 
T&E  team  to  assess  the  operational  feasibility  of  implementation 
of  DOTS  type  system  in  a "real-world"  environment  and  using 
existing  operational  personnel. 

The  original  Preliminary  User's  Test  Guide  was  significantly 
expanded  to  be  used  in  support  of  this  training  as  well  as  the 
subsequent  T&E.  This  final  User's  Test  Guide**  included  sections  on  the 
mechanical  operation  of  the  models,  diagnostic  validation  tests  for 


8Des1qn  of  Training  Systems,  User's  Test  Guide.  TAEG,  30  May  1975. 
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Test  and  Evaluation  (Cont'd) 


each  model  to  permit  revalidation,  and  projected  cost  factors  pertinent 
to  using  the  models. 

• Training  of  the  T&E  Team 

As  previously  indicated,  the  team  was  provided  information  in  the  form  of 
a preliminary  guide  and  goals.  In  addition,  prior  to  formal  T&E  training, 
the  team  members  were  provided  comprehensive  guidelines  by  the  team  leader. 
These  guidelines  will  be  summarized  in  the  next  sub-section.  The  team 
entered  the  formal  training  session  at  Norfolk  with  an  understanding 
of  these  documents. 

During  formal  sessions,  the  T&E  team  members  were  given  executive, 
analyst,  and  some  operator  level  training.  This  broader  scope  was 
included  to  permit  a more  valid  assessment  of  the  operational  consid- 
erations to  be  evaluated. 

• T&E  Team  Model  Analysis 

The  week  following  T&E  training  was  devoted  to  analysis  at  Norfolk. 

The  IBM  team  was  available  for  technical  assistance  during  this  week. 

The  T&E  team  analysis  was  organized  around  three  major  assessment 
areas.  These  were  operational,  technical,  and  financial  feasibility 
of  the  DOTS  models.  It  was  understood  that  the  main  emphasis  of  the  week 
of  analysis  would  be  on  operational  feasibility,  although  Initial 
data  for  the  technical  and  financial  areas  would  be  identified.  ^ 

To  accomplish  the  above,  the  T&E  team  was  divided  into  five  dis- 
cussion areas,  each  managed  by  a leader.  These  areas  and  their 
sub-areas  were  as  follows: 

Potential  Model  Contributions 


At  the  activity  level 
At  other  levels 

Ease  and  Practicality  of  Use 

User  knowledge  requirements 
Update  requirements 
Response  time 
Output  form 

Data  Requirements 

Nature  of  the  data  utilized  by  models 
Data  availability 
Data  accessibility 
Data  management 
Future  data  problems 
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Organizational  Implementation 
Prospective  levels  of  application 

Relationship  of  models  to  existing  policies  and  structures 
Implementation  support 

User  Investment 


Personnel  resources 
Hardware 

Method  of  operation 

The  T&E  team  completed  their  analysis  week  with  a preliminary  posi- 
tion that  was  refined  and  expanded  during  the  follow-on  analysis  to 
be  described  in  the  next  sub-section. 

• Functional  Visitation 

Following  the  week  of  analysis  at  Norfolk,  and  extending  to  September, 
the  various  members  of  the  T&E  team  and  TAEG  conducted  a number  of 

functional  visitations  and  reviews  pertinent  to  the  preliminary 

results  of  the  June  analysis. 

Final  documentation^  was  completed  during  this  period.  A summary 
and  selected  excerpts  from  this  final  report  are  covered  in 

the  next  SECTION  III.  The  T&E  Report  should  be  considered  as  one  of 

the  major  products  of  the  three-phase  DOTS'  effort  since  it  repre- 
sents an  assessment  of  the  practical  value  of  mathematical  modeling 
techniques  to  the  NAVEDTRACOM. 

In  order  to  accomplish  the  T&E  at  Norfolk,  it  was  necessary  to  provide  the 
Norfolk  FLETRACEN  with  access  to  a computer.  A Command  and  Control  Center 
(CCC)  was  installed  consisting  of  an  alpha/numeric  display  terminal  and 
printer  linked  via  dial-up  telephone  lines  to  a commercial  time-sharing 
computer  service.  National  CSS,  Inc.,  located  in  Norwalk,  Connecticut.  NCSS 
provided  the  following  equipment  at  Norfolk: 

• One  Hazel  tine  2000  Video  Display  Terminal.  The  Hazel  tine  2000 
displays  25  lines  of  80  characters  each  and  operates  at  a 30  cps 
transmission  rate. 

• One  Hazel  tine  Impact  Printer  Model  300.  The  Model  300  prints  at 
30  cps,  has  a carriage  width  of  118  characters,  and  employs  a 
tractor-type  paper  feed. 

• One  Hazeltine  Dual  Magnetic  Tape  Cassette  Unit,  having  a 400  bpi 
storage  density  and  a capacity  of  approximately  125,000  characters. 

The  Cassette  Unit,  which  can  operate  in  either  an  off-line  or  on- 
line mode,  serves  as  a storage  unit  for  later  display  of  data  on 

the  Video  Display  Terminal  and  for  editing  text  inputs  to  the  system. 

9feC0Kug;  septSni^!ra1n1nq  Systei,ls- Test  and  Eva1uat1nn  Fina1 
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The  CCC  configuration  was  intended  to  provide  a full  complement  of  terminal 
capabilities  for  evaluation  of  model  usability  in  an  operational  environment. 
The  NCSS  accounting  system  also  provided  specific  costs  for  each  interaction 
as  input  to  the  financial  feasibility  assessment  of  the  T&E. 

In  summary,  the  T&E  was  directed  toward  the  assessment  of  a wide  range  of 
considerations,  as  opposed  to  simple  verification  of  model  design.  As 
previously  indicated,  it  could  be  assumed  that  models  could  be  identified, 
designed,  and  developed  that  would  work.  The  key  question  had  to  do  with 
the  practicality  of  models  in  the  unique  NAVEDTRACOM  environment. 

The  T&E  team  did  not  assess  the  various  sub-task  results,  Phases  I and  II, 
leading  to  test  model  selection  or  general  recommendations.  The  T&E  Final 
Report  specifically  excludes  the  following: 

• Comprehensive  functional  study  of  NAVEDTRACOM. 

• Strategic  assumptions  for  the  1980  decade. 

• Delineation  of  technological  gaps  in  the  training  system. 

• Development  of  the  Educational  Technology  Assessment  Model  (ETAM). 

• Audio-Visual  aids  for  orienting  Navy  training  managers  to  the 
nature  and  use  of  management  science  techniques. 

The  first  three  items  are  covered  in  the  Phase  I Final  Report;  the  fourth 
(ETAM)  is  currently  under  development;  the  fifth  is  a set  of  tape/slide 
presentations  providing  an  orientation  tutorial  pertinent  to  the  three 
DOTS  models. 
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SECTION  III 


TEST  AND  EVALUATION  RESULTS 


TEST  AND  EVALUATION  SUMMARY 

The  purpose  of  this  section  is  to  provide  a summary  of  the  DOTS1  Test  and 
Evaluation,  Final  Report10  and  IBM  comments  pertinent  to  the  T&E's  positions. 

A summary  may  highlight  key  conclusions  but  it  cannot  substitute  for  an  in-depth 
review  of  the  actual  report.  Although  every  attempt  was  made  to  avoid  bias 
in  this  summarization,  it  is  possible  that  the  intended  emphasis  was  inadvertently 
shifted.  As  appropriate,  direct  quotes  from  the  T&E  Report  will  be  included.  In 
every  case,  these  quotes  are  outlined  in  a separate  box. 

The  T&E  Final  Report  is  organized  as  follows: 

• Part  One  - Summary  of  Results  and  Recommendations 

• Part  Tv/o  - Technical  Feasibility 

• Part  Three  - Operational  Feasibility 
t Part  Four  - Financial  Feasibility 

• Appendices  - A through  G 

Generally,  this  summary  will  follow  the  same  format.  Abbreviated  summary 
statements  will  be  followed  by  selected  quotes  and  the  quotes  by  a commentary. 
Report  page  numbers  are  provided. 

All  members  of  the  IBM  technical  team  reviewed  the  T&E  Report  and  were  given 
an  opportunity  to  respond  to  its  conclusions.  Generally,  there  was  agreement 
with  the  conclusions  and  commendation  for  the  report's  thoroughness  and  pro- 
fessional ism. 

The  chart  on  the  next  page  provides  a summary  of  the  T&E  Statements  as  well  as 
SECTION  III  page  locations. 


l°DiGialleonardo,  F.,  Design  of  Training  Systems,  Test  and  Evaluation  Final  Report, 
N P RDCT  'SepeniBe'r^TTT.  


III-l 


TAEG  REPORT  NO.  28 


DESIGN  OF  TRAINING  SYSTEMS 
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1114 

General  Conclusion Considering  technical,  operational  and  financial  factors,  it  is  feasible  to  apply 

mathematical  modeling  techniques  to  the  NAVEDTRACOM  planning  and  decision  process. 

1114 

MI-5 

MI-6 

in-9 

Technical  Feasibility The  three  DOTS  models  are  logically  valid  (function  as  designed)  and  feasible. 

It  was  not  proven  (nor  disproven)  that  they  accurately  simulate  or  predict  actual  situations. 

HI-9 

11140 

Ill'll 

111-17 

Operational  Feasibility- The  DOTS  models  are  operationally  feasible  at  the  schoolhouse  and  func- 

tional command  (COMTRALANT  and  COMTRAPAC)  levels  considering  operational  personnel  required 
for  implementation,  current  management  structure  and  practices,  existing  operational  and  planned 
automated  data  systems,  and  DOTS  data  requirements. 

MI-18 

MI-18 

111-19 

MI-21 

Financial  Feasibility The  three  DOTS  models  were  not  proven  (nor  disproven)  to  be  financially 

feasible,  based  on  either  current  operational  tasks  displaced  or  costs  avoided  through  model  applica- 
tion. Based  on  empirical  observation,  costs  of  implementation  appear  to  be  minimal  and,  therefore, 
could  probably  be  justified  on  the  basis  of  probable  benefits. 

IM-21 

MI-22. 

111-24 

111-25 

Recommendations- While  the  DOTS  effort  appears  to  be  complementary  to  existing  and  planned 

NAVEDTRACOM  applications  of  Automated  Data  Systems  (ADS),  there  is  a need  for  significantly 
increased  management  and  coordination  of  the  composite  ADS  by  CNET.  This  increase  must  take 
place  if  maximum  benefit  is  going  to  be  derived  from  any  data  base  or  modeling  effort. 

111-25 

MI-28 

III  29 

III  30 

Criticism  The  three  DOTS  models  were  not  verified  during  Phase  III. 

MI-30 

111-31 

IM-32 

; n-33 

Criticism  ——Utility  of  the  DOTS  test  models  has  not  been  proven. 

MI-33 

MI-33 

11-34 

111-35 

Recommendation  - Maximum  utility  of  mathematical  models  can  only  be  achieved  through  applica- 

tion to  higher  command  levels  than  that  represented  by  the  Norfolk  Fleet  Training  Center. 

IM-35 

IM-38 

111-39 

111-39 

Criticism  The  Phase  II  benefit/cost  analysis  is  not  valid. 

111-39 

11143 

111-44 

III  44 

Recommendation  There  is  a need  for  establishment  of  a NAVEDTRACOM-wide  program,  under 

direction  of  a CNET  agent,  devoted  to  the  development  and  coordination  of  management  science 
techniques. 

11144 

11145 

ill-46 

11146 

Criticism The  cost  of  developing  the  three  test  models  appears  excessive. 

11146 

11146 
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• General  Conclusion  (Part  1 - T&E) 

Statement  - General  Conclusion 

Considering  technical,  operational  and  financial  factors,  it  is 
feasible  to  apply  mathematical  modeling  techniques  to  the  NAVEDTRACOM 
planning  and  decision  process. 

Quotes:  (T&E  Report) 

Page  5 ..  .It  is  the  conclusion  of  this  analyst  that  the  DOTS 
models  have  established  beyond  reasonable  doubt  the  feasibility 
of  applying  modeling  techniques  to  Navy  training  management 
problems.  This  general  finding  is  a composite  of  the  elements 
of  technical,  operational,  and  economic  feasibility .. . 

Page  6 ..  .The  SCRR  model  demonstrates  the  technical  feasibility 
of  providing  training  center  managers  with  more  accurate  and 
multi-dimensional  assessments  of  their  training  capabilities .. . 

...In  its  current  non-statistical  form,  the  TPF  has  shown  that 
it  can  be  a useful  tool  in  the  hands  of  training  managers  by 
taking  full  advantage  of  presently  available  data  and  certain 
operationally  utilized,  if  still  ill-defined , planning  factors 
extant  in  the  current  training  environment... 

Page  ...No  particular  faults  could  be  found  with  the  composi- 
tion of  the  ETE.  Its  primary  problem  is  that  while  none  of  the 
models  were  strictly  field  tested,  the  ETE  was  the  least  tested 
of  the  lot . . . 

...The  results  of  this  validation , while  thoroughly  documented, 
do  not  provide  insights  to  how  useful  the  model  can  be  to  actual 
users . . . 

...This  resulted  in  substantial  proof  that  the  ETE,  despite  its 
generality , could  perform  in  superior  fashion  to  a system 
specific  model  in  a given  case.  It  does  not  tell  anything 
about  whether  there  will  be  much  opportunity  for  use  of  such  a 
model  given  the  nature  of  the  EDTRACOM's  operations.  A more 
meaningful  test  of  the  appropriateness  of  the  ETE  to  current 
or  future  Navy  training  problems  was  apparently  limited  by  the 
unavailability  of  appropriate  subject  courses... 

Page  10  ...Opportunities  for  applying  the  models  at  the  TRACOM 
level  look  exceedingly  good.  Both  COMTRALANT  and  COMTRAPAC 
seem  receptive  to  application  of  the  models  in  their  respective 
organizations . . . 

Page  50  ...The  potential  for  contribution  by  the  DOTS  models 
in  the  overall  planning  area  is  its  most  significant  feature... 
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Statement  - General  Conclusion  (Cont'd) 

...DOTS  is  envisioned  as  a possible  device  to  effect  the 
changes  required  in  these  various  systems  to  provide  a truly 
reliable  and  accurate  MIS  with  the  added  ability  to  model... 

...If  the  DOTS  concept  did  nothing  more  than  precipitate  an 
investigation  and  realignment  of  the  current  morass  of  Train- 
ing Command  MIS(s)  it  would  be  of  value.  Given  that  it  will 
provide  capabilities  to  model  and  simulate  and  therefore  pro- 
ject as  well,  it  is  potentially  of  great  value... 


Commentary: 

The  quotes  above  provide  positive  support  to  the  General  Conclusion  Statement. 
However,  they  should  be  tempered  with  those  included  in  the  Criticism  and 
Recommendation  sub-section,  which  will  follow  the  next  three  sub-sections. 

The  T&E  supported  the  following  conclusions  from  the  DOTS  Phase  II  Final 
Report^ ! : 

Conclusions  (Quote  from  the  Phase  II  Final  Report) 

The  Naval  Education  and  Training  System  can  improve  the  effectiveness 
of  its  decisions  pertinent  to  resource  planning  and  control  through 
use  of  computerized  mathematical  models... 

The  SCRR,  ETE,  and  TPF  models  are  logically  valid  and  do  perform 
as  designed. . . 

Sufficient  historical  and  operational  data  are  available  in  existing 
records  to  enable  operational  implementation  on  the  DOTS'  models... 

As  will  be  repeatedly  stressed  in  this  SECTION,  the  T&E  team  was  charged  with 
assessing  the  DOTS'  test  models  as  they  apply  to  the  current  NAVEDTRACOM  deci- 
sion and  planning  process.  The  T&E  team's  endorsement  should  not  be  extrapo- 
lated to  include  the  DOTS  Phase  II  conclusion  that,  to  gain  maximum  benefit 
from  mathematical  modeling,  changes  will  be  required  in  the  decision  and  plan- 
ning process.  The  Phase  II  Final  Report  stated,  "This  can  be  accomplished 
with  reasonable  changes  to  the  current  management  system  and  practices." 
SECTION  IV  of  this  report  will  address  the  question  of  change. 

Also,  the  endorsement  of  model  logic  and  design  should  not  be  interpreted  as 
applying  to  model  fidelity.  The  T&E  team  did  not  feel  that  the  ability  of  the 
models  to  accurately  simulate  or  predict  actual  events  had  been  proven  or 
disproven.  The  subject  of  historical  verification  will  be  covered  in  the  next 
sub-section  concerning  Technical  Feasibility. 


!! Design  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2,  Vol . 1, 
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Statement  - General  Conclusion  - Commentary  (Cont'd) 


The  T&E  team  approached  its  evaluation  on  the  assumption  the  three  DOTS'  test 
models  would  be  applied  within  the  existing  planning  and  decision  structure, 
procedures,  and  motivational  environment.  The  original  test  model  and  bed 
selections  were  made  on  the  assumption  that  the  existing  structure,  procedures 
and  motivational  environment  would  be  subjected  to  "reasonable  change."  This 
dichotomy  should  be  considered  in  reviewing  the  T&E  results. 


ii* 
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• Technical  Feasibility  (Part  2 - T&E) 

Statement  - Technical  Feasibility 

The  three  DOTS  models  are  logically  valid  (function  as  designed) 
and  feasible.  It  was  not  proven  (nor  disproven)  that  they 
accurately  simulate  or  predict  actual  situations. 


Quotes:  (T&E  Final  Report) 


SCRR 


Page  16  ...The  SCRR  was  developed  to  meet  four  specific 
types  of  management  problems.  These  are  discussed  below... 

. . .Assessment  of_  io£2Zl££E  training  demand . . .What  it  can 
do  is  take  near  or  long-term  training  demand  projections 
as  input,  and  then  operate  to  determine  whether  such  require- 
ments can  be  feasibly  met  given  current  or  projected  resources 
and  procedures.  In  this  sense,  given  sufficient  lead  time, 
it  can  aid  in  training  resource  planning  and  capabilities 
assessment . . . 

Page  1 7 . . ‘Assessment  of  short-term  demand  fluctuation  arising 
from  unscheduled  events  (e.g. , a_  Ship  repaip  operation.,  reserve 
activation , unusual  seasonal  recruitment  levels) . Here 
different  resource  and  demand  configurations  can  be  tried  out 
to  assess  the  impact  of  such  perturbations , minimizing  their 
dysfunctional  effects  and  maximizing  the  responsiveness  of 
training,  through  better  allocation  of  available  resources... 

. . .Assessment  of  training  resource  utilisation  . . .Using 
current  demand,  SCRR  can  compute  an  optimum  resource  combina- 
tion . . . 

. . .Comparison  of  alternative  resource  allocation  ptratepie^.  • • 
This  is  a case  of  the  model  in  which  multiple  strategies  are 
evaluated,  given  known  performance  and  requirements  character- 
istics. This  is  one  of  the  best  used  of  the  linear  program- 
ming capabilities  of  the  models  and  could  precipitate  substan- 
tive changes  in  present  procedures  for  developing  plans  and 
programs  as  well  as  justifying  and  revising  the  same... 

Page  21  . . .Model  Validation  . . .The  basic  LP  technology  and 

the  MPSX  software  are,  of  course,  not  in  need  of  validation 
due  to  their  standardized  nature.  The  validation  documented 
by  the  model  developer  (TAEG  Report  12-2,  Vol . II,  pp  11-30-60) 
demonstrate  that  the  model  is  functioning  properly.  The 
model's  ability  to  accurately  represent  the  real  world 
is  not  demonstrated.  This  must  be  done  with  historical  data 
or  under  field  test  conditions.  In  the  absence  of  such 
information,  the  SCRR  is  judged  to  be  a useful  and  practical 
representation  of  the  training  complex... 
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Statement  - Technical  Feasibility  (Cont'd) 

...This  judgment  is  a function  of  ( 1 ) feedback  from  pro- 
spective users  obtained  in  the  user  TSE  and  other  discus- 
sions, (2)  a comparison  of  the  model 's  formulation  with 
a description  of  the  training  complex  operation... 

TPF 

Page  22  ...The  objective  of  the  TPF  is  to  provide  a means  for 
simulating  the  flow  of  students  through  a training  complex. . . 

...Given  inputs  such  as  maximum  or  average  course  capacity, 
number  of  convenings,  and  demand,  the  TPF  computes  course 
utilization,  backlog,  AOB,  etc.,  showing  these  on  a quar- 
terly basis  through  time... 

...The  variables  currently  dealt  with  by  the  TPF  have  been 
screened  from  a larger  set  of  variables  on  the  basis  of 
statistical  analyses.  An  original  objective  of  the  TPF 
was  to  develop  statistical  parameters  from  these  analyses 
which  could  be  included  in  the  model  for  the  purpose  of 
predicting  training  results  as  a function  of  student 
profile  data.  Correlations  resulting  from  these  statis- 
tical analyses  were  not  of  sufficient  significance  to 
warrant  inclusion  in  the  model  and  therefore  were  not 
incorporated . . . 

...the  TPF,  as  a simulation,  mathematically  represents  the 
mechanics  of  the  training  center  scheduling  function  from  a 
deterministic  rather  than  probabilistic  basis... 

...The  magnitude  of  original  potential  payoff  from  this 
model  is  substantially  reduced  as  a result  of  its  present 
inability  to  statistically  relate  student  profile  data  to 
student  behavior  within  the  training  system  and  the  conse- 
quent inability  to  gauge  the  potential  impact  of  such 
factors . . . 

Page  24  . . .student  profile  data  and  the  results  of  the 
statistical  analysis  of  that  kind  of  data  are  not  incor- 
porated into  the  model . . . 

...the  idea  of  formulating  a model  to  predict  impact  on  the 
training  process  from  various  types  of  student  inputs  remains 
intuitively  desirable... 

Page  25  . . .Model  Validation . The  validation  procedures  for 
the  TPF  as  described  by  the  model  developer  (TAEG  Report  12-2, 
Vol.  II,  IV,  pp  22-30)  do  not  constitute  a true  validation. 

The  model  is  simply  exercised  in  a number  of  conditions  and 
its  deterministic  processes  are  allowed  to  respond  accordingly. 
It  is  clearly  demonstrated  that  the  model  is  in  operating  order. 
However,  there  is  no  basis  for  judging  how  well  the  model 
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Statement  - Technical  Feasibility  (Cont'd) 


simulates  the  real  training  process  flow.  All  that  can  be 
said  is  that  the  factors  utilized  in  the  TPF  appear  to  be 
reasonable  since  they  are  drawn  from  those  which  are  commonly 
used  by  the  training  managers... 

...Thus  we  know  that  the  TPF  is  a useful  automated  version  of 
the  kind  of  scheduling  that  is  presently  being  carried  out 
at  the  FTC.  We  have  a more  flexible  and  quick  means  for 
developing  schedules  and  assessing  demand  impacts.  It  has 
not  been  demonstrated  that  we  have  a more  precise  or  accurate 
means  for  making  such  developments  or  assessments... 

ETE 

Page  26  ...The  ETE  is  an  entity  flow  simulation  model  for 
ILS  course  design  and  management.  Like  the  TPF  and  SCRR, 
this  model  is  intended  to  answer  questions  concerning 
capabilities  (capacity) , utilization,  and  resource  require- 
ments as  well  as  course  completion  time.  It  has  the  capability 
to  evaluate,  in  compressed  time,  the  effects  of  real  or  pro- 
jected changes  in  resources  or  demand.  Since  equipment  and 
personnel  can  be  costed  external  to  the  model,  the  model  can 
be  used  in  making  cost-effectiveness  determinations  of  dif- 
ferent ILS  course  designs... 

Page  27  ...The  validation  was  not  in  terms  of  how  well  the 
ETE  simulation  represented  actual  course  behavior , but 
rather  to  the  extent  that  results  replicated  those  obtained 
from  the  application  of  a previously  developed  special 
purpose  simulation  model  of  the  EW  school . . . 

...it  is  only  known  that  the  ETE  is  as  good  as  or  better  than 
another  simulation  model  as  applied  to  the  EW  school  use.  Nor 
does  this  validation  procedure  establish  the  generality  of  the 
ETE  as  implied  by  the  model  developer  (TAEG  Report  12-2, 

Vol . II,  pp  23-24) . . . 

...With  the  exception  of  the  model  validation  procedure 
(which  may  have  been  a necessity)  the  ETE  appears  to  have 
been  soundly  developed  from  a technical  viewpoint . However, 
this  technical  soundness  is  as  of  yet,  not  bulwarked  by  a 
clearly  demonstrated  utility  (i.e.,  a utility  determination 
arrived  at  through  interaction  with  actual  ILS  designers  and 
managers) . . . 


General 


Page  28  ...In  general,  the  quality  of  the  validation  proce- 
dures for  the  DOTS  models  conducted  during  both  model  develop- 
ment and  the  T&E,  suffers  from  the  lack  of  a legitimate  field 
test . . . 
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Statement  - Technical  Feasibility  (Cont'd) 


. . .Present  conditions  for  field  testing  of  the  SCRR  and  TPF 
appear  to  be  very  good  at  both  the  COMTRALANT  and  COMTRAPAC 
sites.  Furthermore , prospects  for  usable  end  products 
emerging  from  such  testing  are  sizable. . . 


Commentary: 

The  three  DOTS  models  were  designed  and  developed  by  three  individuals.  Each  re- 
viewed the  T&E  Final  Report  and,  although  there  was  limited  disagreement  on  some 
of  the  Report's  minor  positions,  the  modelers  felt  the  key  points  and  conclusions 
were  accurate  and  thoroughly  assessed. 

As  was  indicated  in  the  T&E  Report,  the  Team  evaluated  the  models  against  their 
application  within  the  current  organization  and  procedural  process  for  planning 
and  control  of  resources  and  requirements.  Although  the  majority  of  the  T&E  was 
concentrated  on  a pragmatic  analysis  of  the  models  as  operational  tools,  some 
attention  was  given  to  their  assessment  as  subsets  to  test  the  DOTS'  recommenda- 
tions evolving  from  Phases  I and  Il12.  The  following  quotes  are  presented  in 
the  Commentary  sub-section  since,  to  a certain  degree,  they  address  those  DOTS' 
recommendations  pertinent  to  higher  levels  of  models  and  institutional  change. 


Quotes:  (T&E  Final  Report) 


Page  7_  . . .The  SCRR  and  TPF  models  were  developed  in  a fleet 
training  environment.  Here  the  demand  is  characteristically 
unprogrammed.  It  is  reasonable  to  believe  that  in  other 
Navy  training  sectors  where  demand  is  much  more  programmable 
(e.g.,  technical  training) , the  potential  for  effective 
model  development  is  at  least  as  great  if  not  greater  than 
has  been  demonstrated  for  the  fleet  environment... 

Page  ...The  models  themselves  neither  require  nor  warrant 
such  incipient  aggrandizement.  They  are  good  examples  of 
how  basic  management  science  technologies  can  be  utilized 
to  assist  in  the  solution  of  specific  management  problems. 

As  such,  they  imply  that  establishment  of  a program, 
chartered  to  utilize  management  science  techniques  in  the 
support  of  Navy  training  management  would  be  well  founded. 
Such  a program  should  not  operate  on  the  basic  assumption 
that  there  is  a total  integrated  solution  to  the  training 
management  problem,  but  rather  on  the  objective  of  achieving 
incremental  improvements  in  management  effectiveness  by  pro- 
viding useful  solutions  to  important  management  problems , 
regardless  of  their  limited  generality... 
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Statement  - Technical  Feasibility  - Commentary  (Cont'd) 


Page  32.. .In  addition  to  more  effective  performance  of  the 
training  management  function  as  currently  practiced , the 
models  have  the  potential  for  introducing  new  management 
techniques  and  supporting  certain  types  of  decision  making 
not  currently  practiced.  The  DOTS  developments  are  seen 
as  contributing  in  two  distinct  but  interdependent  modes. 
First,  as_  a_  management  information  system  by  virtue  of 
the  data  base  management  capabilities  developed  to  support 
the  models.  This  was  not  a direct  objective  of  DOTS  but 
was  a prerequisite  to  producing  models  which  can  effectively 
function  in  the  operational  environment.  Second,  as  a 
management  decision  suyyort  system  to  be  used  in  the  kinds 
of  functions  noted  above.  More  specific  evaluation  of 
potential  model  contributions  is  presented  further  below. . . 

Page  36  .. .Several  participants  viewed  the  bulk  of  model 
potential  to  lie  in  application  at  the  "functional"  level 
and  above  (i.e.,  "models  and  necessary  support  may  not  be 
worth  it  at  schoolhouse  level").  On  the  other  hand, 
participants  with  "schoolhouse"  level  experience  expressed 
the  view  that  considerable  opportunity  exists  to  utilize 
model  potential,  given  "a  little  managerial  imagination." 

A concise  assessment  of  actual  contribution  can  only  be 
obtained  via  a field  test  (which  FTC,  Norfolk,  was  not)... 

. . .A  possible  hindrance  to  vertical  expansion  of  model 
application  is  the  plethora  of  overlapping  management 
systems  which  reside  at  the  upper  management  levels.  While 
the  integration  of  these  systems  is  not  within  the  scope 
of  DOTS,  higher  level  implementation  of  DOTS  development 
may  be  jeopardized  nevertheless.  Coping  with  this 
dilemma  remains  a future  challenge  to  the  DOTS  project 
management . . . 

Page  43  . . .DOTS  is  not  primarily  an  information  collecting/ 
collating  system,  but  instead  provides  optimization  of 
information  output  from  other  systems  (i.e.,  NITRAS) . The 
three  DOTS  models  evaluated  should  be  viewed  as  subsystems 
within  a TRACOM  MIS. . . 


Within  the  boundaries  of  the  T&E,  technical  feasibility  was  established.  As 
indicated  previously,  this  statement  should  be  tempered  with  those  included 
in  the  Criticism  and  Recommendation  which  concludes  this  SECTION. 

Except  as  reflected  in  the  quotes  above,  it  should  be  emphasized  that  the 
T&E  did  not  assess  the  implications  of  an  integrated  management  system,  nor 
was  the  team  tasked  with  this  responsibility.  Therefore,  endorsement  of 
technical  feasibility  by  the  T&E  should  not  be  interpreted  as  including  all 
of  the  main  points  of  the  DOTS  Phase  II  Conclusions  and  Recoranendations. 
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Statement  - Operational  Feasibility 

The  DOTS  models  are  operationally  feasible  at  the  schoolhouse  and 
functional  command  (COMTRALANT  and  COMTRAPAC)  levels  considering 
operational  personnel  required  for  implementation,  current  manage- 
ment structure  and  practices,  existing  operational  and  planned 
automated  data  systems,  and  DOTS  data  requirements. 

Quotes:  (T&E  Final  Report) 

SCRR  - TPF 

Page  6 . . . (SCRR) . . .Like  the  TPF,  it  takes  advantage  of 
existing  planning  factors  such  as  the  student/ instructor 
ratios  which  reflect  basic  requirements  formulas  in  order 
to  determine  organizational  requirements  on  the  basis  of 
the  interaction  of  resources  and  capabilities . When  more 
precise  planning  factors  become  available  in  the  future 
(e.g. , through  the  SHORSTAMPS  program),  they  can  be 
easily  assimilated  into  the  model... 

...The  SCRR,  in  a sense,  is  an  automation  of  what  is 
currently  being  done  mentally  by  instructors  and  training 
managers.  This  is  viewed  as  a plus.  Using  the  SCRR, 
within  the  course  of  an  hour  it  might  be  possible  to 
develop  as  many  resource  allocation  alternatives  as  a 
training  manager  could  develop  in  two  weeks.  This 
advantage,  of  course,  would  be  multiplied  many  times 
over  considering  frequency  of  occurrence.  Add  to  this 
the  capability  for  optimizing  across  resource  allocation 
alternatives  and  one  can  see  that  the  SCRR  presents  the 
potential  for  some  very  real  and  immediate  payoffs  for 
the  training  manager... 

Page  7_  . . .The  SCRR  and  TPF  models  were  developed  to  be 
consistent  with  data  currently  being  collected.  Inputs 
to  the  NITRAS  data  base  (which  are  supplied  to  CNET)  were 
used  to  comprise  a common  data  base  for  the  two  models. 

Certain  additional  data  elements  were  collected  directly 
from  the  FTC;  however,  these  comprise  only  a small  portion 
of  the  total  data  base.  All  of  the  data  used  by  these 
models  are  both  available^  and  accessible  within  CNET. 

About  90%  of  the  data  must  be  collected  for  reporting  to  the 
NITRAS,  regardless  of  model  requirements.  The  situation  is 
not  just  peculiar  to  FTC,  Norfolk,  but  is  true  throughout 
the  CNET  commands . . . 

Page  8 ...In  general,  members  of  the  User  T&E  team  viewed 
the  models  as  operationally  feasible  at  the  TRACOM  and 
schoolhouse  level.  Additional  feedback  later  obtained  from 
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COMTRALANT  and  COMTRAPAC  reinforced  this  finding.  However, 
feedback  obtained  from  CNTT  claimed  that  the  operational 
feasibility  and  desirability  of  the  models  had  not  been 
demonstrated  at  Norfolk. . . 

Page  £ . . .Phases  II  and  III  of  DOTS  have  shown  that  meaning- 
ful and  useful  models  can  be  constructed  for  the  benefit  of 
training  management.  It  has  also  revealed  that  the  most 
effective  residence  for  such  models  in  fleet  training  may 
be  at  the  TRACOM  level . . . 

...In  no  way  can  this  line  of  reasoning  be  extended  to  say 
that  the  models  thus  far  developed  are  appropriate  to  CNTT's 
most  critical  management  problems... 

Page  10  . . .Both  commands  (TRALANT  and  TRAPAC)  have  the 
necessary  equipment  for  supporting  interactive  management 
models. . . 

...Update  requirements  for  the  data  base  were  not  excessive... 

. . .Actual  operation  of  the  models  was  possible  with  a minimum 
of  training.  The  interactive  nature  of  the  model  and  data 
base  programming  contributed  highly  to  the  ease  of  use.  Ap- 
proximately two  weeks  would  be  a reasonable  time  for  training 
an  operator  using  basic  skill  types  presently  available  on 
the  organization's  staff... 

...Considerably  more  time  would  be  required  to  fully  educate 
training  managers  in  potential  uses  of  the  model , if  for  no 
other  reason  than  that  it  would  be  best  to  extend  such  train- 
ing overtime  thereby  allowing  real  and  current  problems  to 
serve  as  examples  in  the  training. . . 

...The  present  documentation  is  satisfactory  and  should  serve 
well  in  future  utilization  of  the  models.  Though  a user's 
guide  has  been  documented,  there  still  remains  a need  for  an 
operator's  manual  which  fully  explains  model  options,  error 
messages  and  corrective  measures.  It  is  understood  that 
such  a manual  is  presently  being  prepared  by  IBM... 

Page  12  ...The  time-sharing  cost  per  usage  of  each  model 
is  well  within  reason  (User  Test  Guide  Appendix  V,  30  May  '75). 
Time-sharing,  of  course,  obviates  the  need  for  large  initial 
investments  in  ADP  hardware,  thereby  putting  the  models  on 
a largely  "pay  as  go"  basis.  It  does  not  appear  that  addi- 
tional staff  would  be  necessary  to  support  the  models  in 
TRACOM  operation.  Modifications  should  not  prove  extensive 
for  TRACOM  application.  All  in  all,  these  factors  point  to 
a very  low  cost  for  implementation  and  operation  in  the  pro- 
jected TRACOM  application.  As  a result,  it  would  not  require 
very  much  "benefit"  to  break  even  in  such  a project... 
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Page  16  ...(SCRR)... The  model  could  be  run  in  a planning  mode 
under  a number  of  different  demand  assumptions . A range  of 
training  capabilities  and  resultant  impacts  could  thus  be 
assessed . . . 

. . .Another  product  of  the  SCRR  relevant  to  this  aspect  is 
the  "optimized  convening  rate. " This  is  the  number  of 
course  convenings  per  course  that  yields  maximum  student 
throughput .. .This  Optimal  Convening  Rate  would  seem  to  have 
very  limited  utility... 

Page  17  ...(SCRR)... This  usage  seems  quite  desirable  given  the 
existence  of  frequent  perturbations  in  demand.  Limited  usage 
at  FTC,  Norfolk,  seems  to  indicate  that  a rather  minimal  level 
of  model  analysis  can  yield  Information  of  broad  and  recurring 
utility  to  the  training  manager.  Yet  usage  of  the  models  by 
managers  seems  to  need  at  least  initial  stimulation... 

...The  decision  to  develop  an  LP  model  to  assess  capabilities 
and  allocate  resources  in  the  training  environment  was  sound... 

Page  19  . . .Any  misconceptions  that  a "requirement"  model 
introduced  to  this  environment  provides  better  basic  require- 
ments information  than  previously  available , must  be  carefully 
avoided.  Yet,  in  the  particular  case  of  the  SCRR,  there  is 
little  doubt  that  modeling  has  demonstrated  the  potential 
for  increasing  the  quality  of  management  information  at  the 
FTC  as  well  as  for  the  functional  command  level  above  it... 

Page  20  ...The  segmentation  of  resources  and  requirements 
within  the  SCRR  limits  its  ability  to  optimize  across  the 
training  center.  While  this  convention  enables  most 
candidate  FTC  problems  to  be  fit  within  reasonable  computer 
storage  limitations , it  appears  to  be  a less  than  accurate 
representation  of  what  is  possible  within  the  real  training 
environment . It  may  be  that  some  of  the  greatest  management 
payoffs  within  the  FTC  are  to  be  gained  from  optimizing 
resources  and  requirements  across  schools... 

Page  22  ...(TPF)... This  simulation  is  intended  to  enable 
managers  to  assess  the  effects  of  changes  in  the  primary 
variables  of  the  training  complex  and  its  immediate  environ- 
ment. The  model  operates  on  two  categories  of  data,  those 
pertaining  to  students  (10)  and  those  pertaining  to  courses 

(15)... 

...Straightforward  proportionality  figures  are  currently  used 
by  the  TPF  to  assess  the  effects  upon  the  training  system  of 
changes  in  demand,  scheduling,  or  capacities.  Failure  rate, 
no-shows,  disenrollments , etc.,  are  not  presently  forecasted 
by  the  model  on  the  basis  of  student  profile  data... 
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...The  choice  of  a process  flow  model  was  a good,  if  perhaps 
obvious,  selection  (and  as  such  did  not  depend  greatly  on  the 
results  of  Phase  I) . The  course  scheduling  problem  is  one 
which  both  lends  itself  to  modeling  and  at  the  same  time 
represents  a significant  training  management  problem.  The 
flow  also  offers  a complementary  capability  to  the  resource 
allocation  model  (SCRR)  that  was  concurrently  developed... 

Page  23  . . .Another  restrictive  factor  is  the  apparently 
limited  opportunity  which  training  managers  at  the  FTC 
level  have  to  utilize  such  a model  in  a proactive  manner. 

This  mostly  stems  from  the  fact  that  fleet  training  demands 
are  at  the  same  time  hard  to  predict  and  hard  to  deny.  On 
the  other  hand,  the  more  reactive  capabilities  of  the  model 
(e.g.,  to  answer  "what  if"  questions ) do  not  appear  to  be 
in  high  demand  at  the  FTC  level,  though  they  may  be  peri- 
odically of  substantial  value  to  that  level.  It  is  important 
in  considering  this  to  keep  in  mind  the  DOTS  models  were 
selected  on  the  basis  demonstrating  the  feasibility  of  an 
approach  for  CNET  and  not  on  the  basis  of  what  models/ 
techniques  would  yield  the  highest  payoff  to  FTC,  Norfolk, 
in  particular... 

...(SCRR  & TPF) . . .answers  from  either  model  would  be 
partially  naive  without  those  of  the  other.  This  prompts 
the  question  of  why  select  and  separately  develop  two  models, 
neither  of  which  is  sufficient  by  itself,  rather  than  develop 
one  integrated  model  from  the  beginning.  The  answer  to  this 
question,  I believe,  lies  in  the  initial  orientation  of  the 
project  -it  has  been  technology  oriented  rather  than  problem 
oriented  (i.e.,  "test  the  feasibility  of  applying  new  decision- 
making technologies" ) . Developing  two  separate  models  pro- 
vided a more  secure  approach  to  that  problem.  Furthermore, 
the  original  concept  of  the  TPF  (i.e.,  to  forecast  training 
complex  impact  as  a function  of  student  profile  data)  lent 
it  more  viability  as  a separate  model  concept... 

Page  36  . . .A  major  consideration  in  the  realization  of 
model  potential  was  determined  to  be  the  level  and  manner 
of  model  use.  While  operation  of  the  model  mechanisms  per  se 
(e.g.,  terminal  operation)  is  not  complex,  use  of  model  results 
requires  all  of  the  skills  of  the  experienced  training  manager. 
Typical  model  support  requirements  (e.g.,  documentation,  ease 
of  use)  were  viewed  as  essential  to  potential  realization. 

(See  "Ease  of  Use"  and  Organizational  Implementation"  selec- 
tions that  follow.) .. . 

Page  37  .. .Specifically , DOTS  presents  a way  for  effectively 
utilizing  the  vast  NITRAS  data  base  in  a proactive  way... 

Pa9e  38  • . .Training  in  use  of  the  DOTS  model  will  be  needed 
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at  all  training  command  levels,  though  to  varying  degrees. 
Essentially  three  levels  of  training  are  required.  These  are 
familiari zation  for  upper  management,  sy&tem  anabasis  for 
translating  upper  management  or  command  training  problems 
into  model  parameters,  and  a systems  operation  for  train- 
ing those  who  would  operate  hardware  components  of  DOTS. . . 

...In  most  cases  (except  at  center  configuration  manage- 
ment level),  system  analysis  and  system  operation  would 
be  collateral  duty  for  the  personnel  involved... 

Page  40  ...It  was  difficult  to  evaluate  this  area  (Data 
Base  Update  Requirements)  because  sufficient  new  problems 
were  not  used  in  the  TSE.  Program  (model)  updating/ 
modification  would  probably  be  rare  for  routine  (local 
command)  report  generation.  A more  extensive  program 
update  requirement  would  probably  exist  at  upper  manage- 
ment levels  for  resolving  projected  ("what  if")  problems... 

Page  42  ...Basic  data  collection/verification/input  to 
the  master  data  base  would  be  responsibilities  of  local 
commands  (centers) . These  procedures  are  more  acceptable 
than  with  other  current  information  because  local  commands 
would  have  direct  access  to  central  system.  These  com- 
mands would  also  have  to  manage  problem-solving  require- 
ments unique  to  the  command  or  center... 

...The  TSE  Team  did  not  think  the  programs  and  data  base 
associated  with  the  SCRRSTPF  models  would  be  frequently 
updated  in  the  operational  phase... 

Page  43  ...The  TSE  Team  believed  that  neither  master  data 
base  nor  program  modifications  would  be  difficult  if 
proper  justification  for  making  these  modifications  could 
be  demonstrated  (through  findings  of  RSD  or  training 
effectiveness  evaluations) . . . 

...Inasmuch  as  DOTS  is  primarily  a projection/management 
system  which  provides  information  for  "what  if"  questions, 
its  use  as  an  information/ report-generating  system  would 
be  redundant,  and  would  represent  gross  underusage  of 
the  system. . . 

...If  used  as  an  information  gathering  system  only,  DOTS 
would  not  substantially  improve  on  response  times  associ- 
ated with  other  available  systems... 

Page  46  ...The  core  of  the  data  base  is  NITRAS  data  which, 
as  "raw  data,"  is  relatively  valid,  reliable  and  accurate... 

(IBM  Note:  Exceptions  were  noted  in  the  T&E  Report.) 
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...The  data  are  presently  available , largely  without  addi- 
tional cost . . . 

...The  data  is  not  presently  either  easily  or  regularly 
accessible...  (From  the  DOTS  Data  Base  - extraction  pro- 
grams required .) 

...Data  base  is  manipulatable  in  a very  impressive  way; 
however^,  the  system  for  accomplishing  this  manipulation 
is  not  Navy  owned  and  is  equipment  specific .. . 

Page  47  ...The  usability  of  data  to  be  accessed  by  the 
DOTS  family  of  models  relates  directly  to  the  NITRAS/NAVTIS 
problems;  such  problems  as  validity , accessibility , avail- 
ability, etc.,  hinge  on  NITRAS  and  other  systems  (e.g., 
SHORSTAMPS  SHOROC , etc.)  either  operational  or  under 
development  of  being  implemented . The  DOT  models  must  be 
viewed  as  a capability/application  consistent  with  the 
total  Training  Command  MIS  and  not  as  a separate  system... 

ETE 

Page  26  ...The  ETE  is  a general  purpose  simulation  model  and 
is  thus  not  structured  around  a specific  type  of  course.  Its 
data  base  would  not  normally  even  approach  the  complexity 
of  that  of  the  TPF  and  SCRR.  The  data  for  a typical  model 
run  can  be  entered  by  the  user  at  the  start  of  a model 
session  (and  then  stored  for  later  recall  or  modification) . 
Theoretically , the  ETE  could  at  present  be  applied  to  any 
ILS  course  or  group  of  courses,  regardless  of  location  or 
even  if  the  course  is  yet  to  be  convened... 

. . .Factors  favoring  the  development  of  this  model  included 
the  following:  (1)  increasing  numbers  of  ILS  courses  through- 
out EDTRACOM,  (2)  increasing  resource  constraints , (3)  exis- 
tence of  previous  work  in  simulating  ILS  in  the  Navy,  (4) 
prospects  for  a general  purpose  model  that  could  immediately 
be  put  to  use  throughout  EDTRACOM , (5)  emphasis  of  other 
models  on  training  process,  as  opposed  to  course  design... 

Page  36  ...This  problem  area  currently  exists  to  a major 
degree  in  CNTT  and  will  do  so  in  the  future  at  the  TRACOMS. 
Present  methods  are  seen  as  inadequate  most  of  the  time 
with  substantial  to  great  opportunity  for  improvement 
(million  dollar  area) . Moderate  to  substantial  changes  in 
present  organization/policies  are  seen  as  necessary  to 
effect  improvements... 

Page  7 ...and  theoretically  has  applicability  wherever  ILS 
courses  are  designed , modified , or  where  a significant  number 
of  such  courses  must  be  managed  given  some  commonality  of 
resource  requirements . . . 
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Page  51  ...While  the  ETE  and  highly  conceptual  ETAM  models 
would  appear  to  have  the  least  impact  external  to  the  Command, 
they  might  have  the  greatest  impact  within.  The  magnitude 
of  the  individualization  effort  within  CNTT  alone,  and  the 
projected  savings  from  this  effort,  would  support  continued 
RSD  of  these  models... 

Page  42  . . .Although  the  data  base  of  the  ETE  model  might 
require  frequent  update,  this  should  not  present  much  of 
a management  problem  because  the  master  data  base  would  not 
be  involved  (only  scratch  disc) . The  ETE  model  per  se  would 
probably  not  be  frequently  updated... 

Page  8 ...The  data  required  for  the  ETE  are  fairly  simple  in 
comparison  to  those  needed  by  the  other  two  models  and  their 
supply  seems  to  present  no  operational  problem. . . 

Page  26  . . .All  of  the  major  ILS  decision  variables  can  be 
represented  and  considerable  user  flexibility  is  provided. 

The  ETE  is  generally  formulated  so  that  the  user  can  formulate 
a specific  model  to  represent  a given  ILS . The  estimated 
time  for  such  formulation  (3-5  days)  is  acceptable... 

General 

Page  49  ...Specific  level (s)  and  degree  of  application  of 
the  DOTS  models  cannot  be  defined  at  this  time.  The  considera 
tions  which  preclude  such  definition  are  essentially  the  lack 
of  costing  information  and  lack  of  experience  based  on  applica 
tion  beyond  FTC,  Norfolk. . . 

Page  52  ...The  group's  consensus  was  that  no  additional  per- 
sonnel  would  be  needed.  That  while  some  training  could  be 
necessary , present  staff  can  learn  to  use  the  models/results .. 

...This  variable  (additional  personnel)  is  largely  unknown  at 
this  point.  It  is,  however,  directly  related  to  use  which  is 
a function  of  the  acceptance  and  utility  of  the  models  them- 
selves. . . 

...Many  existing  hardware  configurations  now  in  place  at 
potential  user  sites  could  be  used  for  DOTS.  The  minimum 
requirements  (i .e . , keyboards/printers)  would  suffice  in  many 
cases.  CRTS  and  other  more  sophisticated  peripherals  would 
not  be  necessary  in  many  locations.  In  a good  many  cases, 
communicating  magnetic  card  typewriters  could  fulfill  all 
the  hardware  requirements  for  access  to  the  DOTS  system, 
without  significant  impact  on  present  use  of  this  equipment. 
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Commentary: 

The  above  quotes  support  the  position  that  the  three  models  are  operationally 
feasible,  considering  the  current  NAVEDTRACOM  organization,  personnel,  resources 
and  decision-planning  practices.  This  support  is  qualified  by  a T&E  concern  for 
the  lack  of  an  operational  verification  during  Phase  III.  In  short,  under  opera- 
tional conditions,  do  the  models  predict  or  simulate  actual  events  accurately 
and,  equally  important,  do  they  have  significant  utility  to  the  appropriate  users? 
This  concern  will  be  addressed  under  Criticism  and  Recommendations. 

The  above  position  represents  a T&E  majority  opinion,  but  it  should  be  noted 
that  the  Naval  Technical  Training  Command  took  serious  exception  to  it.  Their 
position  will  be  stated  in  the  CRITICISM  AND  RECOMMENDATIONS  sub-section. 
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Statement  - Financial  Feasibility 

The  three  DOTS'  models  were  not  proven  (nor  disproven)  to  be  finan- 
cially feasible,  based  on  either  current  operational  tasks  displaced 
or  costs  avoided  through  model  application. 

Based  on  empirical  observation,  costs  of  implementation  appear 
to  be  minimal  and,  therefore,  could  probably  be  justified  on  the 
basis  of  probable  benefits. 

Quotes:  (T&E  Final  Report) 


Page  12  . . .As  for  proving  the  financial  feasibility  of  applying 
modeling  techniques  in  general  throughout  the  EDTRACOM,  it  is 
not  believed  that  this  could  presently  be  established. . . 

Page  1 1 . . .Numerous  attempts  were  made  during  the  T&E  to  trace 
back  from  the  application  of  the  models  to  real  problems 
emanating  from  the  FTC  and,  in  turn,  to  assess  the  benefits 
of  supporting  the  solution  of  those  problems.  None  of  these 
attempts  resulted  in  very  useful  information. . . 

Page  87  ...On  the  benefit  side,  perhaps  the  most  meaningful 
indicator  revealed  in  the  T&E  was  provided  by  comments  from 
the  User  T&E  team  that  the  modeling  and  MIS  capabilities 
demonstrated  by  DOTS  would  be  worth  at  least  two  billets  at 
the  training  center  level  of  implementation.  As  the  technical 
feasibility  section  of  this  report  pointed  out,  a field  test 
is  necessary  to  precisely  identify  a measure  of  benefit. 
However,  the  costs  as  estimated  above  are  low  enough  to  make 
application  of  the  models  a low-risk  proposition,  especially 
at  the  TRACOM  level . . . 

...The  greater  question  of  financial  feasibility  concerns  to 
what  extent  the  results  of  DOTS  can  be  extrapolated  to  assess 
the  feasibility  of  applying  such  management  techniques  through- 
out the  EDTRACOM  in  general . Since  the  techniques  employed 
by  the  DOTS  models  (e.g.,  UP,  simulation)  are  a fair  sample  of 
the  field  from  which  they  are  drawn,  it  seems  reasonable  to 
generalize  the  order  of  operating  costs  estimated  above... 

Page  12. . .Applications  can  differ  so  markedly  with  respect 
to  cost  and  payoffs  that  any  specific  application  hardly 
provides  a basis  for  extrapolation  of  feasibility  for  the 
general  case.  The  history  of  management  science  is  strewn 
with  examples  of  very  good  general  models  which  never  were 
implemented.  Situational  factors  for  each  case  seem  to 
override  any  generalizations  that  might  be  made... 

Pa9e  — '"With  the  dearth  of  information  on  the  benefit 
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side  of  the  feasibility  question,  it  is  best  to  focus  on 
cost  factors . Here  it  can  be  seen  that  the  future  cost 
of  developing  and  operating  the  models  may  prove  to  be 
effectively  less  than  even  assessed  by  the  contractor  in 
his  analysis . This  position  can  be  supported  because  of 
two  concurrent  developments  to  DOTS.  First,  the  data 
base  necessary  for  future  model  development  is,  in  the  main, 
being  compiled  regardless  of  DOTS.  Thus,  the  cost  of  that 
data  acquisition  and  maintenance  need  not  be  attributed  to 
the  DOTS  models.  Secondly,  the  TRACOMS  either  have  or  are 
in  the  process  of  procuring  the  necessary  hardware  to  sup- 
port DOTS  type  models,  again  regardless  of  DOTS.  For 
example,  TRAPAC  already  has  in  operation  a terminal  with 
full  telecommunications  capability  from  which  the  models 
have  already  been  accessed.  Thus,  it  is  doubtful  that  users 
would  need  to  purchase  equipment  that  they  would  not  have 
procured  otherwise . . . 

Page  49  .. .Throughout  the  TSE  we  have  been  hampered  by  a 
lack  of  costing  data.  We  are  unable  to  predict,  with  any 
degree  of  validity,  what  this  capability  might  cost  in  any 
of  its  terms  (hardware , software,  personnel,  telecommunica- 
tions, site  prep.)  much  less  as  a whole.  This  is  not  a 
criticism  in  that  derivation  of  systems  costs  was  not  a goal 
of  the  effort  to  this  point,  but  rather  a statement  of  fact. 
This  fact  does  serve  to  highlight  the  other  issue,  however, 
in  that  while  activity  level  usage  might  be  great  in  terms  of 
numbers  of  interactions,  does  the  utility  warrant  the  invest- 
ment? We  cannot  say  at  this  time.  The  most  we  can  say  is 
that  if  continued  evaluation  is  directed , that  a portion 
of  the  evaluation  must  address  the  economics  of  the  system 
and  that  the  economic  evaluation  must  address  alternative 
methods  of  application  as  well  as  levels... 

Page  53  . . .Given  finding  in  first  (hardware  configuration) 
and  fourth  (number  of  ultimate  weeks)  discussion  areas , what 
might  be  a.  "ball park"  cost  per  incidence  of  usage  ratio . 

$20K  a year  estimate  by  IBM  seems  excessive  if  hardware  is 
largely  in  place... 

Page  86  ...As  a result  of  discussions  with  members  of  the 
user  TSE  team  and  visits  to  various  EDTRACOM  sites,  it  can 
be  validly  assumed  that  utilization  of  computer-based  models 
like  those  of  the  DOTS  project  will  not  require  special  data 
or  equipment  acquisitions  of  a significant  nature.  From 
this  it  can  also  be  assumed  that  the  basic  skills  needed  to 
operate  the  models  will  also  be  present  at  most  sites.  Since 
model  operation  and  data  maintenance  which  does  not  replace 
existing  procedures  is  estimated  to  consume  a very  small 
number  of  manhours,  it  is  not  anticipated  that  support 
personnel  cost  will  be  appreciable.  There  will  be  a train- 
ing cost.  On  the  basis  of  training  conducted  prior  to  the 
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on-site  user  T&E,  this  is  estimated  at  about  four  man-weeks 
per  site  (2-4K) . Data  pertaining  to  time-sharing  cost  for 
the  models  were  obtained  from  actual  model  operation  at  FTC, 
Norfolk  (see  TAEG  Report  30  May  1975,  User's  Test  Guide, 

Section  V) . Using  these  cost  data  and  a somewhat  arbitrary 
estimate  of  the  extent  of  usage,  an  estimate  of  $10K  per 
year  was  obtained  (using  all  three  models) . With  some 
amortization  of  training  cost,  this  results  in  a total 
yearly  cost  that  can  be  safely  estimated  at  under  $15K, 
allowing  for  some  equipment  depreciation  (or  rental  cost 
allocation)  and  model  maintenance... 

Page  49  . . .Discussions  regarding  systems  costs  and  economy 
of  operation  centered,  in  the  main,  on  whether  or  not  school- 
house  level  implementation  could  ever  be  shown  to  be  econom- 
ically feasible  or  even  justifiable.  This  argument  centered 
on  the  premise  that  the  unavailability  of  precise  costing  data 
notwithstanding,  the  thrust  of  implementation  should  be  toward 
the  functional  level  in  that  only  at  this  level  would  the 
magnitude  and  complexity  of  problems  warrant  the  capability 
to  apply  modeling  techniques... 

. . .Counter-arguments  stated  the  premise  that  costs  would,  in 
all  likelihood , be  minimal  to  the  point  that  they  would  not 
be  the  primary  deterrent  to  systems  implementation;  and,  further, 
that  there  were  many  ways  in  which  the  models  could  (should) 
be  used  at  the  activity  level.  That  the  amount  of  usage  and 
attendant  benefits  to  be  derived  from  such  use  is  limited  only 
by  the  imagination  and  resourcefulness  of  the  activity  itself. . . 


Commentary: 

The  IBM  technical  team  agrees  with  the  T&E  Report: 
Quote  (T&E  Final  Report): 


Page  11  ...It  appears  that  a reasonable  benefit/cost  analysis 
of  the  models  can  only  be  derived  from  a valid  field  test.  It 
must  be  established  to  what  extent  the  models  supplant  existing 
procedures  or  personnel  and  what  additional  capabilities  are 
provided  - even  more  importantly  what  these  capabilities  buy 
in  the  way  of  greater  effectiveness... 


This  conclusion  supports  one  of  the  recommendations  evolving  from  a hypothetical 
benefit/cost  feasibility  exercise,  based  on  theoretical  assumptions,  conducted 
during  Phase  II.  That  recommendation  was: 
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Quote  (Phase  II  Final  Report):^ 


A decision  to  implement  the  DOTS  models  at  the  eighteen  user 
locations  identified  in  the  preceding  Projected  Operational 
Responsibility  sub-section,  should  be  based  on  a thorough 
cost  versus  savings  and  avoidance  justification.  That 
justification  and  decision  should  not  be  made  prior  to  the 
completion  of  DOTS  Phase  III  and  an  assessment  of  actual 
operating  results  at  the  Norfolk  Fleet  Training  Center  test 
site. 

The  Phase  II  DOTS  validation  exercise  did  provide  sufficient 
indicators  for  a preliminary  projection  of  cost  savings  and 
avoidances.  Necessarily,  this  analysis  had  to  be  based 
primarily  on  subjective  opinion  and  assumptions,  due  to  lack 
of  objective  operational  data. 

It  should  be  noted  that  the  referenced  exercise  during  Phase  II  was  based  on 
the  assumption  that  NAVEDTRACOM  management  practices  would  be  modified  to 
achieve  maximum  benefit  from  mathematical  models,  whereas  the  T&E  team  was 
tasked  with  assessing  the  DOTS  models'  financial  implications  as  applied 
in  the  current  environment.  However,  the  Phase  II  recommendation  that  the 
cost  justification  take  place  after  Phase  III  is  equally  valid  for  either 
assumption. 

The  T&E  Final  Report  expressed  serious  concerns  pertinent  to  the  Phase  II  pro- 
jection of  costs  and  avoidances  hypothesized  for  an  integrated  NAVEDTRACOM 
modeling-management  system.  Since  the  T&E  team  specifically  excluded  the 
concept  of  institutional  change  from  its  evaluation,  these  concerns  should 
not  be  considered  as  pertinent  to  this  statement  of  financial  feasibility. 
However,  the  T&E  team  provided  strong  support  for  the  Phase  II  recommenda- 
tion that  a definitive  cost  avoidance  exercise  take  place  after  Phase  III. 

The  T&E  team's  concerns  will  be  addressed  in  more  detail  under  the  CRITICISMS 
AND  RECOMMENDATIONS  sub-section  to  follow. 


13Design  of  Training  Systems,  Phase  II  Final  Report.  TAEG  Report  No. 
Vol.  I,  Section  IV,  December  1974,  Page  IV-19. 


12-2, 


I 11-22 


TAEG  REPORT  NO.  28 


• Criticisms  and  Recommendations  (Parts  1-4,  T&E) 

The  T&E  Final  Report  contained  a very  comprehensive  technical 
critique  of  the  three  models.  This  sub-section  will  not 
attempt  to  duplicate  or  summarize  that  technical  material  but 
will  emphasize  those  criticisms  and  recommendations  pertinent  to 
the  general  DOTS'  conclusions  and  long-range  implications. 

Elimination  of  the  technical  critique  does  not  imply  disagree- 
ment. The  IBM  technical  team  felt  the  technical  critiques  and 
recommendations  were  valid  and  should  be  considered  in  any 
future  DOTS  effort. 

The  format  used  in  this  sub-section  is  similar  to  that  of  previous 
ones.  A statement,  based  on  a consolidation  of  comments  distributed 
through  the  T&E  Report,  is  presented,  followed  by  the  actual  quotes 
and  that  by  an  IBM  commentary.  Each  quote  is  coded  as  follows: 

0 - Observation 

C - Criticism 

R - Recommendation 

As  previously  indicated,  consolidation  of  dispersed  quotes  to 
support  any  given  position  statement  can  introduce  a certain 
degree  of  bias.  However,  as  long  as  the  reader  is  aware  of  this 
approach  and  relies  on  the  T&E  Report  for  a more  detailed  analysis, 
this  summary  should  highlight  the  key  T&E  concerns  and  recommenda- 
tions. 
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Statement  #1  - Criticisms  and  Recommendations 

While  the  DOTS'  effort  appears  to  be  complementary  to  existing  and  planned 
NAVEDTRACOM  applications  of  Automated  Data  Systems  (ADS),  there  is  a need  for 
significantly  increased  management  and  coordination  of  the  composite  ADS  by 
CNET.  This  increase  must  take  place  if  maximum  benefit  is  going  to  be  derived 
from  any  data  base  or  modeling  effort. 


Quotes  (T&E  Final  Report): 


Page  47  (0) 

. . .From  specific  and  general  comments  by  attendees  to  this 
TSE,  a strong  display  of  feeling  was  evident  that  the  EDTRA 
Command  is  in  dire  need  of  management  in  the  area  of  systems 
development  and  coordination.  Specifically , there  is  the 
current  problem  of  a lack  of  coordination  of  systems  develop- 
ment and  inter f acting . There  is  a requirement  for  an  overall 
information  systems  management/development  function  that  is 
independent  from  the  operational  administration  of  the  ADP 
facilities  in  the  EDTRACOM . Some  form  of  information  systems 
study/development  function  has  to  exist  that  addresses  the 
needs  of  all  levels  of  the  Command,  not  just  CNET... 

Page  41  (R) 

...Update/modification  requirements  in  the  case  of  upper 
management  may  center  around  problems  involved  in  integrating 
DOTS  with  other  information  systems.  A serious  effort  should 
be  undertaken  to  determine  complementary  functions  among 
these  information  systems  if  redundancies  are  to  be  minimized 
and  efficient  integration  and  upper  management  usage  are  to 
be  attained.  This  effort  is  not  part  of  the  current  DOTS 
mission  or  tasking. . . 

Page  37  (0) 

. . .A  recurrent  theme  of  the  TSE  discussions  centered  on  the 
NITRAS  data  system  (i.e . , its  shortcomings) . The  complementary 
characteristics  of  the  DOTS  data  base  management  system  was 
recognized . . . 

Page  14  (R) 

...The  results  of  the  DOTS  project,  primarily  the  application 
of  data  base  management  technology,  should  be  integrated  more 
closely  with  the  work  of  the  NETISA  organization  in  the  develop- 
ment of  the  NITRAS  data  base... 

Page  50  (0) 

...The  central  point  is  that  DOTS  is  not  to  be  another  system 
separate  and  distinct  from  those  already  in  being.  It  must 
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not  be  competitive.  It  must  be  complementary  to  a total  MIS 
established  from  the  myriad  of  (unintegrated)  systems  that 
now  exists.  Similarly , DOTS  (as  a function  of  the  total 
Training  Command  MIS)  must  interface  with  those  systems  ex- 
ternal to  the  Training  Command , such  as  SHORSTAMPS , that  will 
impact  on  the  Command . . . 

Page  87  (0) 

. . .However,  a plus  factor  for  products  such  as  the  DOTS  models 
is  the  current  state  of  management  information  systems  (MIS) 
within  the  EDTRACOM . These  are  currently  emergent  and  embryon- 
ic. Since  the  models  are  very  complementary  to  the  MIS  develop- 
ment process,  a fairly  consistent  MIS  benefit  might  be  projected 
from  one  development  site  to  another.  This  underlying  benefit 
might  be  capable  of  covering  most  risks  associated  with  the 
more  sophisticated  utilities  of  models... 

Page  50  (0) 

...The  need  for  accurate,  timely  planning  is  well  recognized 
and  there  are  several  systems  now  operational  or  under  opera- 
tional development  and  implementation  which  purport  to  provide 
this  capability  (none  appear  totally  satisfactory  be  it  a 
function  of  scope  of  concept  or  of  performance) . . . 


Commentary  - Statement  #1 : 

The  above  quotes  emphasize  a major  concern  of  NAVEDTRACOM  planners  at  all  levels. 
That  concern  has  to  do  with  the  lack  of  timely  and  accurate  data  required  to  re- 
spond to  both  operational  and  planning  requirements.  A related  complaint,  fre- 
quently voiced,  was  that  lower  levels  provided  significant  amounts  of  data  to 
higher  ones  but  were  unable  to  benefit  from  that  effort  through  subsequent  re- 
trieval . 

Although  specifically  excluding  an  evaluation  of  the  DOTS  Phase  I effort  from  its 
assessment,  the  T&E  team  conclusions  did  support  one  of  that  Phase's  major  recom- 
mendations. The  recommendation  can  best  be  described  through  the  following  random 
quotes  extracted  from  the  Phase  I Final  Report: 

Quote  (Phase  I Final  Report):^ 

Page  VI -328  ...It  is  important  that  methods  be  available  to 
track  key  attributes  of  the  system.  For  example,  report 
accuracy  and  timeliness  were  observed  to  be  a problem  in 
a number  of  instances;  however,  there  did  not  appear  to  be 


^Design  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 
December  1973. 
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any  pressing  management  concern  since  backup  systems  were 
available.  In  the  ideal  system  for  managing  training 
resources,  quality,  reliability/availability,  timeliness, 
and  usefulness  of  data  would  not  only  be  available  to 
measure  the  system,  but  they  would  also  be  integral  to 
objective  setting  and  measurement  of  training  managers. 

Also,  before  the  system  effectiveness  can  be  evaluated 
in  terms  of  its  role  as  a part  of  the  overall  resource 
management  system,  these  data  must  be  available... 

Page  VI-341  ...Closely  related  to  the  preceding  items  is 
the  need  for  faster  access  to  resource  data.  This  capability 
is  especially  important  for  detecting  and  analyzing  variance, 
costing  training  plans,  responding  to  queries,  etc... 

Page  V 1-340  ...Resource  management  reports  need  to  be  made 
more  timely,  more  accurate,  and  more  relevant  to  the  command 
level  at  which  they  are  used.  Performance  reporting  (actual 
vs  plan)  should  have  increased  emphasis  in  terms  of  automated 
and  timely  feedback  to  reduce  the  reliance  on  manual  logs 
and  analyses. . . 

Page  VI-304  ...Several  locations  were  noted  to  have  systemat- 
ically defined  their  data  needs,  others  were  in  the  process  of 
defining  systems  to  make  more  effective  use  of  the  existing 
data  bases  and  to  reduce  the  amount  of  manual  activity  in 
the  preparation  of  reports... In  an  idealized  training  system 
there  should  be  greater  standardization  and  integration  of 
methods,  procedures,  and  equipments  used  in  the  development 
and  processing  of  management  information... 

Page  VI -302  ...Resource  and  technical  data  must  be  supplied 
to  the  various  systems  throughout  the  reporting  period  so 
that  meaningful  data  can  be  reported  back  to  the  activity 
for  control  purposes... a substantial  amount  of  manual  activity 
was  noted.  This  was  apparently  due  to  the  timing  and  accuracy 
of  the  basic  reporting  documents.  It  would  seem  from  the 
comments  that  there  are  excessive  errors  contained  in  the 
input  data  which  promulgates  throughout  the  system.  Thus, 
there  is  a real  question  as  to  how  good  a job  is  done  to 
verify  the  reasonableness  and/or  accuracy  of  input  data... 
However,  an  important  aspect  of  increasing  the  effectiveness 
of  resource  control  is  to  have  resource  managers  assured 
that  their  reports  are  highly  accurate... 

Page  VI -343  ...Paperwork.  It  appears  that  a high  percentage 
of  the  manual  activity  devoted  to  preparing  basic  documents 
could  be  more  fruitfully  applied  in  the  analysis  and  solution 
of  resource  management  problems.  This,  of  course,  implies 
either  the  elimination  or  the  automation  of  much  of  it. 

Some  of  both  appears  desirable... 
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Page  VI-74  ...Another  facet  of  an  ideal  system  is  consis- 
tency and  uniformity  of  designating  an  instructional  organiza- 
tion. The  large  difference  in  the  size  of  training  - centers, 
schools,  detachments,  units  - presents  a most  confusing 
picture.  The  origin  of  such  differences  is  understandable 
since  the  present  more  centralized  training  structure  picked 
up  a number  of  relatively  disparate  elements.  However, 
there  is  a need  for  a realignment  of  current  terminology 
and  organizations... 

Page  V I - 1 02  . . .The  idealized  system  requires  optimal  planning. 
Aspects  of  "optimum"  include:  uniformity  in  planning  across 
organizations,  use  of  equal  cost/usage  and  other  factors, 
capability  to  include  contingencies,  and  the  capability  to 
project  into  a future  time  period... An  initial  requirement  is 
a set  of  common  planning  factors  for  use  by  all  elements  of 
the  NETS.  Such  factors  must  be  centrally  developed,  main- 
tained, and  controlled.  In  addition,  uniform  procedures 
for  the  utilization  of  these  factors  must  be  developed  and 
made  mandatory  for  all  organizations... 

Page  V 1-340  ...The  number  of  ways  in  which  resources  are  cate- 
gorized, i .e. , by  program  element,  function  sub-portions, 
budget  classification  code,  element  of  expense,  etc.,  seems 
unduly  complicated  for  the  most  effective  application  of  the 
system  at  all  levels... 

Page  VI-104  ...Factors  should,  as  applicable,  be  expressed 
in  terms  of  ranges  or  sets  with  associated  assumptions  to 
permit  the  planner  to  develop  alternatives  or  to  revise  plans... 

Page  VI-341  ...Realistic  costing  means  the  availability  of 
historical  costs  and  planning  factors  as  well  as  a system  for 
modifying  them  to  the  needs  of  any  new  costing  exercise... 

Page  VI -340  ...A  more  well-defined  accounting  unit  is  needed 
for  planning,  tracking,  and  controlling  training.  Present 
units,  such  as  average  under  instruction,  average  on-board, 
and  student  man-months,  are  subject  to  varying  interpretations 
which  preclude  an  accurate  measurement  of  actual  training... 

Page  VI -340  ...Greater  and  more  uniform  automatic  data 
processing  capability  is  required  to  adequate  service  all 
command  levels.  The  non-uniformity  and  the  absence  of  suf- 
ficient processing  capability,  however,  is  especially  notice- 
able at  the  lower  command  levels... 


These  Phase  I recommendations  and  observations  were  based  on  assessment  of  the 
NAVEDTRACOM  in  support  of  the  overall  DOTS'  thrust  which  envisioned  an  inte- 
grated multi-level  system  of  mathematical  models  operating  in  a modified 
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management  environment.  A Phase  I task  was  to  identify  the  modifications 
required  to  achieve  maximum  benefit  from  such  an  approach.  Lack  of  data  was 
one  of  the  major  institutional  gaps  identified. 

That  same  gap  is  proving  a serious  concern  in  the  day-to-day  planning  process. 
The  IBM  recommendations  in  SECTION  IV  of  this  report  will  amplify  and  support 
the  T&E  Report  in  this  area. 
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The  three  DOTS  models  were  not  verified  during  Phase  III. 

Note:  DOTS'  documentation  defines  validation  as  proving  that  the  models  did 
work  as  designed  and  verification  as  proving  they  accuractely  simulated  or  pre- 
dicted actual  events.  In  the  quotes  to  follow,  verification  should  be  substi- 
tuted for  validation.  The  T&E  did  support  the  Phase  II  validation  but  not 
verification,  as  was  originally  planned  for  Phase  III. 


Quotes  (T&E  Final  Report): 


Page  21  (0  and  C) 

. . . (SCRR) . . .The  basic  LP  technology  and  the  MPSX  software  are, 
of  course,  not  in  need  of  validation  due  to  their  standardized 
nature.  The  validation  documented  by  the  model  developer  (TAEG 
Report  12-2,  Vol.  II,  pp  11-30-60)  demonstrate  that  the  model 
is  functioning  properly.  The  model's  ability  to  accurately 
represent  the  real  world  is  not  demonstrated... 

Page  25  (0  and  C) 

. . . (TPF) . . .The  validation  procedures  for  the  TPF  as  described 
by  the  model  developer  (TAEG  Report  12-2,  Vol.  II,  IV,  pp  22-30) 
do  not  constitute  a true  validation.  The  model  is  simply 
exercised  in  a number  of  conditions  and  its  deterministic 
processes  are  allowed  to  respond  accordingly .. .However , there 
is  no  basis  for  judging  how  well  the  model  simulates  the  real 
training  process  flow.  All  that  can  be  said  is  that  the  factors 
utilized  in  the  TPF  appear  to  be  reasonable  since  they  are  drawn 
from  those  which  are  commonly  used  by  the  training  managers... 

IBM  Note:  The  logic  validation  of  Phase  II  is  confused 

with  the  intent  to  verify  in  Phase  III.  The  referenced 
report  does  contain  a logic  validation  description  which 
did  prove  the  model  operated  as  designed.  The  above 
quote  is  correct  in  indicating  that  no  proof  of  fidelity 
in  simulating  actual  conditions  (verification)  was  provided. 

Page  28  (0) 

...These  conditions  did  not  rule  out  validity  testing  using 
historical  data.  Unfortunately , the  historical  data  available 
would  not  have  been  complete  enough  to  allow  even  this  type  of 
testing. . . 

IBM  Note:  "These  conditions"  refer  to  those  conditions 

precluding  a meaningful  field  test. 


Ill -29 


TAEG  REPORT  NO.  28 


Statement  #2  - Criticisms  and  Recommendations  (Cont'd) 


Page  25  (0) 

. . .a  true  validation  would  have  entailed  the  use  of  historical 
conditions  and  the  matching  of  model  output  to  actual  outcomes 
in  the  training  environment  resulting  from  those  conditions . 
There  appears  to  have  been  no  attempt  at  such  a procedure. 

(Of  course,  this  would  be  more  of  a validation  of  the  planning 
factors  used  by  the  model  than  of  the  mechanics  of  the  model 
per  se.) . . . 

Page  27  (0  and  R) 

...In  the  absence  of  ILS  at  FTC,  Norfolk,  the  ETE  was  validated 
on  the  basis  of  a proposed  consolidated  EW  school  at  Corry 
Station. . . 

...the  ETE  remains  for  the  most  part  in  an  unvalidated  state. 

It  may  be  that,  as  the  model  developer  states,  there  are  no 
NAVEDTRACOM  ILS  applications  that  are  presently  suitable  for 
use  in  validation.  If  this  is  still  true,  the  condition 
should  not  prevail  for  long.  The  ETE  should  be  validated 
against  a number  of  different  actual  ILS  cases  as  soon  as 
possible . . . 


Commentary  - Statement  #2: 


The  T&E  Report  is  correct;  very 
The  original  DOTS  plan  projected 
It  was  to  have  taken  place  in  a 
by  the  IBM  technical  team  on  the 
Cape  Kennedy  Facility.  Data  was 
planned  for  this  task,  and  that 
during  Phase  II  using  real-world 


limited  verification  was  performed  in  Phase  III, 
a major  verification  effort  during  this  Phase, 
laboratory  environment,  i.e.,  verification  runs 
IBM  System/360  Model  40GF  located  at  their 
to  have  been  historical.  Twelve  months  were 
task  was  to  "...verify  the  models  developed 
data  and  operational  situations."'5 


Based  on  an  assessment  of  Phase  I results,  a significant  change  in  approach 
was  indicated  early  in  Phase  II.  There  were  three  factors  indicating  need 
for  change.  One,  there  seemed  to  be  little  need  to  expend  R&D  resources  in 
proving  that  mathematical  models  could  be  designed  which  could  accurately  simu- 
late the  types  of  conditions  reflected  by  Phase  I's  candidate  model  list.  Two, 
only  limited  historical  data  was  available  in  a form  amenable  to  the  proposed 
experimental  verification  effort.  Any  conclusions  based  on  a verification  using 
the  available  data  would  have  been  subject  to  so  many  qualifications  that  its 
value  would  have  been  very  limited.  Three,  the  most  important  questions  had 
to  do  with  operational  feasibility.  Would  models  have  value  when  applied  to 
the  real  world  of  planning  and  decision-making?  If  they  had  value,  could  they 
be  applied  in  the  general  environment  considering  human  factors? 


15Pesign  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 
December  1973,  Page  1-2. 
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Based  on  these  three  considerations,  the  Phase  III  verification  in  a laboratory 
environment  was  changed  to  a feasibility  assessment  of  the  three  test  models 
implemented  in  an  operational  test  bed.  The  number  of  tasks  required  to  in- 
stall the  models  in  a time-sharing  system,  develop  the  required  supporting 
software,  and  to  train  the  test  bed  staff  precluded  also  undertaking  any  signifi 
cant  verification  effort.  To  have  reasonable  validity,  such  a task  would  have 
required  most  of  the  planned  twelve  months. 

In  no  way  does  the  above  contradict  the  T&E  team's  recommendation  for  some  form 
of  verification.  Whether  the  final  decision  is  to  implement  the  three  DOTS' 
test  models  as  is,  modified  versions,  or  totally  different  models,  a historical 
verification  should  take  place  in  a field  test  environment. 

The  next  Statement  addresses  such  a test  for  the  DOTS  models. 
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Utility  of  the  DOTS  test  models  has  not  been  proven. 

Note:  Utility  has  to  do  with  the  frequency  the  models  would  be  applied  to 
current  planning  and  operational  needs  by  a user,  and  the  ultimate  value  of 
that  application. 

Quotes  (T&E  Final  Report): 

Page  57  (C) 

. . . Throughout  the  T&E  there  had  been  a dearth  of  feedback  as 
to  FTC,  Norfolk,  reactions  to  the  specific  utility  of  the 
various  model  information  products,  though  some  general  reactions 
had  been  provided  by  the  FTC  DOT  and  the  TRALANT  T&E  representa- 
tive. It  was  determined  that  while  the  FTC  application  was  not 
a field  test  per  se,  the  level  of  model  development  and  oppor- 
tunity for  application  was  such  that  specific  examples  of  model 
utility  using  actual  (if  historical)  problems,  along  with  FTC 
management  reactions,  should  have  been  produced  in  the  course 
of  DOTS  Phase  III. . . 

Page  28  (0) 

. . .an  issue  for  the  T&E  is  whether  or  not  a field  test  should 
have  been  conducted  in  Phase  III.  It  is  the  judgment  of  this 
T&E  that  such  a field  test  could  not  have  been  comprehensively 
performed  in  Phase  III. 

...It  would  not  have  been  possible  to  obtain  the  user  commit- 
ment necessary  for  proper  field  testing  without  first  being 
able  to  demonstrate  that  the  models  were  in  working  order, 
that  they  could  be  made  to  operate  in  the  training  environment, 
and  that  they  addressed  substantive  training  issues  in  a use- 
ful and  meaningful  manner... 

Page  55  (R) 

...The  next  phase  of  DOTS  should  include  field  test(s)  of 
the  developed  models,  under  operational  conditions... 

Page  12  (R) 

.. .Initiation  of  data  collection  and  review  for  TRALANT/ 

TRAPAC  application  and  field  test  of  SCRR-TPF.  This 
should  include  a clear  assessment  of  opportunities  for 
model  contributions  at  these  commands... 

Page  13  (R) 

. . .Presentation  of  the  ETE  model  to  the  core  of  ILS  course 
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developers  in  the  EDTRACOM.  This  to  be  followed  by 
field  testing  of  the  model  in  several  cases  of  actual 
course  design.  An  assessment  of  the  benefits  of  the  model 
should  be  made  following  these  tests  and  the  model  should 
be  further  developed,  revised,  or  put  on  the  shelf  for 
operational  use  as  appropriate... 

Page  28  (R) 

...This  field  testing  should,  at  a minimum,  fill  in 
some  of  the  blanks  left  from  this  THE  evaluation  regarding 
more  precise  estimates  of  cost,  accuracy,  and  utility.  A 
suitable  field  test  site  for  the  ETE  model  is  not  so 
evident  but  must  be  pursued.  It  would  not  be  wise  to 
put  that  model  on  the  shelf  and  wait  for  1980  to  happen... 


Commentary  - Statement  #3 

The  IBM  technical  team  agrees  that  utility  has  not  been  proven  and  should  be 
demonstrated  through  additional  field  testing. 
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Maximum  utility  of  mathematical  models  can  only  be  achieved  through  applica- 
tion to  higher  command  levels  than  that  represented  by  the  Norfolk  Fleet 
Training  Center. 

Note : The  issue  of  level  was  a major  one  during  the  T&E  with  CNTECHTRA  taking 
the  strongest  position. 

Quotes  (T&E  Report) 

Page  8 (0  and  C) 

...The  essential  pitfall  of  the  models  as  viewed  by 
CNTT  is  that  they  "have  been  primarily  designed  to  help  that 
level  of  management  least  requiring  assistance  in  decision- 
making" (i.e.,  the  schoolhouse) . CNTT  construed  this  view 
as  being  supported  by  the  acting  Director  of  Training  at 
FTC,  Norfolk,  who  reported  to  the  T&E  team  that  the  need 
to  answer  "what  if"  questions  was  rare  (i.e.,  5 or  6 times 
a year)  at  the  FTC  level . (Of  course  the  Director  also  had 
many  good  things  to  say  about  the  models.  He  recommended 
their  continued  development  at  the  TRACOM  level  and  hoped 
that  the  FTC  could  have  continued  access  in  the  models  on 
a periodic  basis.) . . . 

...Comments  on  User  T&E  Results:  While  CNTT's  point  is 

sound  in  its  premise  (i.e.,  that  the  FTC  site  is  atypical  of 
the  greater  EDTRACOM) , it  has  deviated  somewhat  in  its  con- 
clusion (i.e.,  that  the  operational  feasibility  of  modeling 
l&s  not  been  demonstrated) . . . 

Page  42  (R) 

. . .A  functional  flow  chart  should  also  be  developed  for  the 
TRACOM  showing  scope  and  frequency  of  decision-making 
information/ problem  solving  required  at  all  levels,  including 
specific  examples  of  problems  and  methods/flow  used  to  resolve 
these  problems... 

Page  54  (R) 

. . .A  general  agreement  evolved  during  discussion  of  user  re- 
quirements to  the  effect  that  expansion  of  the  prototypical 
modeling  system  to  TRAPAC  and  CNTT  activities  is  justified . 

Such  expansion  would  address  a wider  variety  and  greater  fre- 
quency of  training  decision-making  conditions , explore  the 
interfaces  necessary  to  complement  existing  and  developing 
systems,  expand  the  data  base,  increase  familiarity  with 
modeling  as  a technique,  potentially  improve  collection  of 
related  data,  and  provide  better  indications  of  system  poten- 
tial than  would  an  extensive  effort  to  go  operational  at  this 
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Page  55  (R) 

...The  thrust  of  DOTS  should  be  modified  to  include  the 
investigation  of  applications  at  higher  levels  (i.e.,  beyond 
FTC ) within  CNET... 


Commentary  - Statement  #4 

Statement  #4  is  compatible  with  the  recommendations  of  DOTS  Phases  I and  II. 
The  test  model  selection  process,  based  on  the  Phase  I study,  illustrates  this 
compatibility.  The  twenty-one  potential  test  models  were  rated  against  nine 
criteria,  one  of  which  was: 


Quote  (Phase  I Final  Report):^6 


Page  VII-13-14. . .The  organizational  level,  i.e.,  CNET,  func- 
tional , or  activity,  which  will  benefit  from  the  development 
of  each  candidate  model  was  taken  Into  consideration  in  rating 
this  criterion.  In  some  cases,  a single  model  or  simple 
modifications  of  that  model  may  benefit  several  levels  of 
training  management.  A high  correlation  should  exist  between 
organizational  level  and  savings  potential,  in  that  decisions 
made  at  higher  management  levels  will  generally  impact  a 
larger  portion  of  the  total  training  budget  or  of  the  total 
training  population.  Thus,  models  designed  to  assist  in 
management  decisions  at  several  organizational  levels  received 
the  highest  rating.  Models  to  be  implemented  at  the  CNET  level 
received  the  next  highest  rating,  while  models  to  be  implemented 
at  the  lower  echelon  level  received  the  lowest  rating... 


The  DOTS  Phase  II  report  recommended  an  evolutionary  implementation  of  mathemati- 
cal models  to  encompass  three  general  levels  of  organizational  and/or  modeling 
complexity. 


Quotes  (Phase  II  Report):^ 


Page  IV-7  ...The  long-range  thrust  of  Technical  Development 
Plan  P43-03  (P01A)  is  towards  the  integration  of  computerized 
mathematical  models  into  all  appropriate  areas  of  the  CNET's 


16Pesign  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 
December  1973,  Pages  VII-13-14. 

^Design  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2, 
VolTlTTtecember  1974,  Page  TII-48-. 
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decision  analysis  process.  This  thrust  implies  change  in 
the  way  decisions  are  currently  being  made  if  the  total 
decision  process  is  to  derive  maximum  benefit  from  the 
models.  Therefore,  an  orderly  and  logical  implementation 
of  both  the  models  and  changes  to  the  decision  process  itself 
are  essential  if  the  ultimate  results  are  to  be  accepted 
and  effectively  used... 

...For  purposes  of  estimating  the  resources  required  for  the 
design,  development,  validation,  and  implementation  of  a total 
CNET's  decision  analysis  system,  models  were  arbitrarily 
divided  into  three  levels.  The  hierarchy  was  based  more  on 
the  functional  use  of  the  models  than  on  the  level  of  training 
command  using  the  results  of  the  models,  although  the  two  do 
tend  to  equate... 

...The  SCRR,  ETE,  and  TPF  models  are  examples  of  the  first 
level.  They  are  primarily  concerned  with  the  projection  of 
training  resources  and  student  flow.  The  development  of  a 
multi-level  DOTS  modeling  system  was  initiated  at  the  first 
level,  since  the  horizontal  implementation  of  these  models 
across  the  Naval  Education  and  Training  System  will  require 
standardization  of  a basic  data  base  and  associated  procedures. 
Such  a standard  will  be  essential  to  the  support  of  higher 
level  models.  This  same  advantage  will  apply  to  the  evolution 
from  the  second  to  the  third  level... 


The  above  quotes  should  only  be  interpreted  as  supporting  the  premise  of 
Statement  #4.  The  T&E  Report  took  strong  exception  to  the  Phase  II  hypoth- 
esis that  significant  benefit  could  be  achieved  from  an  integrated  multi- 
level modeling  system  operating  within  a modified  management  environment. 


Quotes  (T&E  Final  Report): 


Page  14  (0) 

...It  is  a conviction  here  that,  at  least  for  the  present , 
the  successful  application  of  management  science  technology 
is  much  more  likely  through  many  small  scale  applications... 


Page  9_  (0  and  C) 

...In  fairness  to  CNTT,  it  should  be  said  that  their 
alarm  that  someone  might  try  to  spread  these  particular  models 
across  the  waters  of  EDTRACOM  is  not  without  cause.  Much  of 
the  DOTS  documentation  concerning  "projected  operations " of 
the  models  invites  this  kind  of  fear  (e.g.,  TAEG  Report  No.  12-2, 
Vol.  I,  Phase  II  Overview,  pp  IV-15  through  IV-19) . Such  pre- 
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sentations  seem  to  imply  that  the  three  models,  with  a minimum 
of  modification,  could  be  applied  in  a very  systematic  manner 
through  much  of  CNET.  This  sort  of  thinking  was  further  embel- 
lished by  verbal  presentations  from  IBM  which  emphasized  that 
there  would  have  to  be  fairly  signiflceuit  changes  in  the  present 
Navy  training  organization  in  order  to  take  full  advantage  of 
the  models. . . 


The  T&E's  concern  for  Phase  II 's  long-range  recommendations  and  hypotheses 
will  be  addressed  in  the  Commentary  for  Statement  #5.  The  CNTECHTRA  concern 
for  the  selection  of  Norfolk  as  a test  bed  is  understandable  since  the  T&E 
approached  the  models  primarily  as  operational  tools,  as  opposed  to  R&D  test 
vehicles.  This  is  not  a criticism  of  the  T&E  team  since  their  operational 
orientation  and  approach  significantly  increased  the  validity  of  the  DOTS 
conclusions  and  recommendations  as  they  apply  to  the  current  environment. 

In  any  event,  the  CNTECHTRA  concern  for  the  selection  of  Norfolk  as  a test 
bed  merits  an  explanation. 

Two  general  principles  guided  definition  of  the  final  test  environment.  These 
were: 

1.  The  environment  must  permit  a reasonable  assessment  of  the  DOTS' 
hypothesis.  Although  only  a limited  subset,  its  evaluation  must 
permit  extrapolation  of  conclusions  to  other  sets  or  subsets  of 
the  system. 

2.  The  test  environment  and  models  selected,  in  addition  to  supporting 
hypothesis  assessment,  must  provide  potential  for  immediate 
practical  return  on  the  R&D  investment.  Test  models,  whether 
functional  or  mathematical,  were  to  be  selected  considering  their 
operational  Implications  if  this  consideration  would  not  be  detri- 
mental to  achievement  of  DOTS'  R&D  objectives. 

The  Norfolk  FLETRACEN  was  selected  during  Phase  II  as  an  operational  test  bed 
for  the  following  reasons: 

- The  FLETRACEN  management  was  agreeable  to  providing  support  for 
the  test  effort. 

- Although  small  compared  to  other  possible  test  locations,  the 
management  activities  and  functions  resident  at  the  Norfolk 
FLETRACEN  represented  all  of  the  test  areas  to  be  explored. 

- Due  to  its  size,  the  additional  data  elements,  beyond  those 
provided  by  NITRAS  and  other  sources,  could  be  developed  within 
the  scope  of  the  DOTS'  effort. 

It  must  be  emphasized  that  the  priority  DOTS'  R&D  objective  was  to  assess 
the  potential  benefit,  if  any,  to  be  derived  from  the  application  of  manage- 
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ment  science,  operations  research  and  mathematical  modeling  techniques  to 
the  total  NAVEDTRACOM  planning  and  decision  process.  To  achieve  this,  a 
limited  subset  was  to  be  modeled  and  the  results  evaluated.  Extrapolations 
were  to  be  drawn  from  this  evaluation.  Recommendations  were  to  be  developed 
which  would  complement  the  Strategic  Assumptions  evolving  from  Phase  I which 
projected  the  NAVEDTRACOM  environment  through  the  mid-1980's. 

It  became  obvious  during  Phase  I that  test  models  could  be  selected  that  would 
serve  as  R&D  test  vehicles  as  well  as  providing  more  immediate  value  to  existing 
operational  management.  However,  the  three  test  models  were  primarily  selected 
based  on  their  potential  for  evaluating  the  original  DOTS'  hypothesis  and  not  for 
their  immediate  value  to  the  Norfolk  FLETRACEN.  To  partially  compensate 
Norfolk  for  its  interest  and  effort  in  supporting  the  test  and  evaluation, 
some  limited  software  was  developed  to  permit  access  to  the  DOTS'  supplemental 
data  base  required  to  drive  the  test  models. 
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Statement  #5  - Criticisms  and  Recommendations 
The  Phase  II  benefit/cost  analysis  is  not  valid. 

Note:  No  benefit/cost  analysis,  appropriate  to  the  application  of  the  three 

test  models  in  today's  management  environment,  was  conducted  by  IBM.  The  T&E 
Report  refers  to  a hypothetical  projection  of  an  integrated  management/modeling 
system  based  on  Phase  I and  II 's  strategic  analysis  and  recommendations. 


Quotes  (T&E  Final  Report): 


Page  11  (C) 

. . .A  benefit/cost  analysis  was  attempted  in  TAEG 
Report  No.  12-2.  A shortcoming  is  its  assumption  that  the 
models  as  presently  configured  could  be  widely  applied 
throughout  CNET.  As  pointed  out  above,  this  premise  must 
be  rejected.  There  are  other  reasons  why  the  benefit/cost 
analysis  provided  by  the  contractor  is  not  acceptable. 

These  will  be  identified  in  the  detailed  T&E  Report... 

Page  86  (C) 

...The  results  of  Phases  II  and  III  in  no  way  warrant 
the  usage  assumptions  made  in  that  analysis.  As  a conse- 
quence, the  resulting  cost  benefit  conclusions  are  not  viewed 
as  acceptable... 

Page  54  (R) 

...If  this  system  progresses  to  a recommendation  for 
operational  implementation,  system  specifics  should  be  left 
to  further  determination  during  the  procurement  process  and 
be  done  in  such  a way  as  to  ensure  investigation  of  all  possible 
configurations  with  attendant  costs... 

Commentary  - Statement  #5 

Notes  from  Pages  11  and  89  of  the  T&E  Final  Report  assume  that  a definitive 
cost  justification  was  attempted  during  Phase  II.  IBM  did  not  attempt  such 
a justification  and  recommended,  at  the  end  of  Phase  II,  that  one  not  be 
attempted  until  after  Phase  III.  The  following  quotes  from  the  Phase  II 
exercise  should  put  it  in  perspective: 


Quotes  (Phase  II  Final  Report):^ 

Page  IV-7  ...The  long-range  thrust  of  Technical  Development 


^Design  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2, 
Vol . 1,  December  1974. 
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Statement  #5  - Criticisms  and  Recommendations  - Commentary  (Cont'd) 


Plan  P43-03 (P01 A)  is  towards  the  integration  of  computerized 
mathematical  models  into  all  appropriate  areas  of  the  CNET's 
decision  analysis  process.  This  thrust  Implies  change  In 
the  way  decisions  are  currently  being  made  If  the  total 
decision  process  is  to  derive  maximum  benefit  from  the 
model s . . . 

Page  IV-9  ...a  forward  pricing  analysis  was  performed  in 
Phase  II,  projecting  the  costs  of  a horizontal  extension 
of  the  SCRR,  ETE,  and  TPF  models  to  form  an  integrated 
CNET  modeling  system... 

...Due  to  the  number  of  variables  and  lack  of  operational 
experience,  this  pricing  exercise  must  be  considered  highly 
speculative.  DOTS'  Phase  III  will  provide  experience  in  an 
operational  environment,  enabling  a more  precise  definition. 

However,  the  projected  costs  presented  here  should  be  suffi- 
ciently accurate  to  support  tactical  planning... 

Page  IV-19  ...A  decision  to  implement  the  DOTS'  models  at  the 
eighteen  user  locations  identified  in  the  preceding  Projected 
Operational  Responsibility  sub-section  should  be  based  on  a 
thorough  cost  versus  savings  and  avoidance  justification. 

That  justification  and  decision  should  not  be  made  prior 
to  the  completion  of  DOTS'  Phase  III  and  an  assessment  of 
actual  operating  results  at  the  Norfolk  Fleet  Training 
Center  test  site... 

Page  IV-21  ...As  previously  stated,  this  exercise  should  be 
repeated  after  more  objective  data  have  been  developed  during 
Phase  III  predictive  validation  of  the  three  models... 

Page  IV-16  ...In  developing  recommendations,  it  was  assumed  that 
the  decision  command  levels  Identified  currently  have,  or  will 
be  granted,  the  authority  to  make  the  training  resource  decisions 
supported  by  the  models... 

The  Phase  II  analysis  was  not  based  on  the  assumption  that  the  test  models  would 
be  implemented  without  design  change  or,  what  is  more  important,  change  to  the 
process  by  which  the  NAVEDTRACOM  manages  resources. 

In  addition  to  developing  and  validating  (did  the  models  work  as  designed)  the 
three  test  models.  Phase  II  was  tasked  with  developing  the  hypothetical  impact 
of  implementing  a multi-level  NAVEDTRACOM  integrated  modeling  system  concurrent 
with  initiating  change  in  the  total  command-management  process  to  achieve  maxi- 
mum benefit.  This  task  was  predicated  on  a Phase  I Strategic  Working  Assump- 
tion (SWA). 
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Statement  #5  - Criticisms  and  Recommendations  - Commentary  (Cont'd) 
Quote  (Phase  I Final  Report):^ 


"A  confluence  of  factors  will  result  in  a significant  reduc- 
tion in  the  resources  available  to  accomplish  the  NETS*  train- 
ing mission.  Although  a current  concern,  the  degree  and  scope 
of  resource  restriction  will  become  increasingly  severe  through 
the  mid-1980's  and  will  result  in  major  changes  to  the  NETS* 
resource  control  approach  and  structure." 

"Economic  necessity  will  dictate  a reduction  in  the  total 
resources  required  to  accomplish  the  training  mission  as  well 
as  a reduction  in  the  unit  cost  of  training  for  any  given  task. 
All  command  levels  of  NETS*  will  experience  increasing  levels 
of  inquiry  and  control  from  higher  government  sources.  The 
following  change  statements  are  pertinent  to  this  reduced 
resource  availability: 

1.  There  will  be  an  increased  capability  for  cost  effective- 
ness quantification,  measurement  and  control. 

2.  More  systematic  and  comprehensive  cost  justification 
methods,  techniques,  programs,  etc.,  will  be  developed 
and  implemented. 

3.  There  will  be  an  improved  capability  for  marginal  cost 
identifi cation. 

4.  Increased  application  of  computer  technology  will  result 
in  a need  for  more  uniform  and  consistent  data  bases." 

♦Note:  "NETS,"  Naval  Education  and  Training  System,  is  now 
currently  designated  as  NAVEDTRACOM. . 


This  SWA  was  based  on  eighteen  Strategic  Assumptions^0  also  developed  and 
analyzed  during  Phase  I.  Phase  II 's  analysis  was  an  attempt  to  make  this 
general  hypothesis  more  specific  and  amenable  to  empircal  analysis.  The  key 
to  the  Phase  II  analysis  is  the  Phase  I assumption  that  severe  resource 
restrictions  "will  result  in  major  changes  to  the  NETS  resource  control 
apprach  and  structure." 


^Design  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1, 

Vol.  1,  December  1973,  Pages  IV-17  and  18. 

^Design  of  Training  Systems,  Phase  I Final  Report,  TAEG  Report  No.  12-1,  Vol.  2, 
December  1973,  Pages  A 11-90  and  91. 
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Statement  #5  - Criticisms  and  Recommendations  - Commentary  (Cont'd) 


The  change  projected  by  the  SWA  was  formulated  as  a recommendation  during 
Phase  II.  This  was  stated  as:21 

"Concurrent  with  model  implementation,  make  the  changes  to 
the  management  system  required  to  permit  effective  Implementa- 
tion of  resource  decisions.  This  implies  granting  various 
levels  of  training  officials  more  authority  over  the  use 
of  training  resources  than  they  now  have." 

The  Phase  II  change  assumption  envisioned  increased  involvement  of  all 
NAVEDTRACOM  commanders  in  the  inspection  and  challenge  of  resource  plans 
and  decisions.  Mathematical  models  were  considered  as  both  management  tools 
to  support  this  change  and  as  change  agents  for  facilitating  its  incorporation. 

To  assist  in  an  initial  cost  speculation,  twenty  potential  users  were  identified 
at  eighteen  geographic  locations22  within  the  NAVEDTRACOM.  Costs  were  extra- 
polated from  Phase  II  experience  with  the  three  test  models. 

The  projected  savings  were  based  primarily  on  the  change  in  management, 
philosophy  and  not  on  the  application  of  the  three  test  models  in  today's 
environment.  Although  it  was  assumed  that  an  integrated  modeling  system 
would  require  a minimum  of  eighteen  users,  this  was  not  the  driving  cost 
avoidance  consideration.  The  driving  hypothesis  was: 

By  increasing  the  degree  of  involvement  by  all  training  command 
levels  in  the  resource  planning  and  decision  process,  and  by  sup- 
porting that  involvement  with  an  integrated  system  of  mathematical 
modeling  tools,  up  to  2.1%  of  annual  specialized  training  costs 
(student  and  NAVEDTRACOM)  can  be  avoided. 

This  2.1%  potential  is  projected  as  the  "best  case"  condition,  and  even  that 
is  not  projected  until  the  fifth  year  after  the  management  change  and  imple- 
mentation of  an  integrated  modeling  system.  Worst  case  projects  .2%  the 
first  year  and  1.5%  in  the  fifth  year,  as  compared  to  a best  case  of  .3% 
the  first  and  2.1%  the  second. 

Based  on  the  more  significant  savings  achieved  in  industrial  training  through 
application  of  similar  techniques,  the  Phase  II  analysis  should  be  considered 
extremely  conservative. 

During  the  T&E,  the  team  expressed  a serious  concern  that  the  Phase  II  hypo- 
thetical analysis  would  be  used  to  justify  implementation  of  an  integrated 
modeling  system  across  the  NAVEDTRACOM  prior  to  a valid  justification.  Their 
concern  is  a reaonable  one  considering  the  number  of  systems  and  educational 
innovations  that  have  been  sold  on  the  basis  of  just  such  hypotheses.  This 
same  concern  was  considered  in  documenting  Phase  II.  The  speculative  nature 


ZlQesign  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2, 
Vol.  1,  December  1974,  Page  V-1. 

22pes  iqn  of  Training  Systems,  Phase  II  Final  Report,  TAEG  Report  No.  12-2, 
Vol.  1,  December  1974,  Page  lV-1^. 
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Statement  #5  - Criticisms  and  Recommendations  - Commentary  (Cont'd) 


of  the  analysis  was  repeatedly  stressed,  as  well  as  the  strong  recommendation 
that  a definitive  cost  justification  take  place  after  Phase  III. 

The  T&E  team  also  expressed  some  concern  for  the  feasibility  of  a change  in 
management  practice.  This  will  be  discussed  in  SECTION  IV,  Conclusions  and 
Recommendations. 
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Statement  #6  - Criticisms  and  Recommendations 

There  is  a need  for  establishment  of  a NAVEDTRACOM-wide  program,  under  direction 
of  a CNET  agent,  devoted  to  the  development  and  coordination  of  management 
science  techniques. 


Quotes  (T&E  Final  Report) 


Page  13  (R) 

...At  the  conclusion  of  the  TRAM  application/ field 
test,  a program  should  be  established  devoted  to  the 
use  and  development  of  management  science  techniques  to 
support  training  management  throughout  CNET.  The  agent 
for  this  program  should  reside  within  the  CNET  community 
but  have  close  ties  with  the  RSD  sector.  This  group  would 
serve  as  a two-way  funnel  for  both  translating  CNET  RSD 
requirements  and  for  implementing  advancements  emanating 
from  RSD.  More  importantly , the  group  would  have  the 
capability  to  directly  address  management  support  require- 
ments as  possible  within  the  state-of-the-art . This  group 
could  provide  an  invaluable  element  of  continuity  and  con- 
sistency in  the  solution  of  such  problems  within  CNET. 

The  DOTS  Phase  I products,  as  corrected  or  amended,  should 
provide  an  excellent  foundation  for  this  program.  The 
models  developed  thus  far  would  be  the  start  of  a "tool  kit" 
of  models  or  other  techniques  that  could  be  applied  to 
problems  as  they  arise  within  the  training  community... 

Page  47  (0) 

...Additionally,  the  strong  possibility  exists  that  by  and 
large  the  systems  now  under  development  or  proposed  are 
essentially  designed  to  serve  only  the  CNET  level  of  need 
and  are  of  little  or  no  utility  to  the  operational  levels 
and  functional  levels  of  the  EDTRACOM , except  in  several 
unique  cases.  Very  little  effort  appears  to  be  directed 
towards  the  development  systems  like  DOTS  which  have  the 
inherent  capacity  to  ljecome  viable  planning  tools 
(i.e.,  for  forecasting,  modeling,  etc.)  or  as  tools 
for  evaluating  various  approaches  to  resource  utiliza- 
tion . . . 


Commentary  - Statement  #6 

The  Phase  I recommendations  provide  strong  support  for  Statement  #6.  Although 
intended  to  apply  to  all  NAVEDTRACOM  R&D  requirements,  the  following  summary 
quote  is  pertinent: 

"The  approach  to  training  research  requires  integration. 

The  current  fragmentary  approaches  do  not  yield  the  best  results 
for  the  expenditure  of  funds.  Central  control  under  the  leading 
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Statement  #6  - Criticisms  and  Recommendations  - Commentary  (Cont'd) 


element  in  naval  training  (CNET)  is  essential.  This  must  include 
systematic  definition  of  problems,  assignment  of  priorities,  and 
prior  commitment  to  implement  effective  results." 

Statement  #6  is  a sound  recommendation.  Additional  reinforcement  is  provided 
on  Pages  VI-362-364  of  the  Phase  I Final  Report. 
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Statement  #7  - Criticisms  and  Recommendations 

The  cost  of  developing  the  three  test  models  appears  excessive. 


Quotes  (T&E  Report):- 
Page  87  (0  and  C) 

...The  costs  incurred  to  develop  the  DOTS  models  appear 
inordinate  and  probably  not  supportable  at  anything  less 
than  command  level  implementation.  But  those  costs  in- 
cluded many  one-time  items  necessary  to  initiation  of 
the  project . . . 

Page  14  (0  and  C) 

...As  a final  remark,  it  should  be  noted  that  while  the 
objective  of  the  DOTS  ADO  is  viewed  as  being  achieved, 
the  magnitude  of  expenditures  that  were  required  should 
be  a lesson  in  the  pitfalls  of  setting  up  projects  which 
are  a priori  tasked  with  pleasing  everybody... 


Commentary  - Statement  #7 

If  DOTS  is  viewed  as  simply  providing  data  processing  solutions  to  current  prob 
lems,  Statement  #7  is  correct.  However,  this  was  not  the  intent  of  the  DOTS 
effort,  and  costs  during  all  three  phases  reflect  its  R&D  objectives. 

The  T&E  team's  comment  appeared  to  be  based  on  the  assumption  that  all  DOTS' 
Phases  II  and  III  expenditures  were  devoted  to  the  selection,  development  and 
validation  of  the  test  model. 

The  actual  test  model  development  took  place  during  Phase  II  along  with  numer- 
ous other  tasks  related  to  the  DOTS  R&D  goals.  Phase  III  was  more  concerned 
with  data  base  development,  data  base  software  support,  command  and  control 
center  installation,  and  training  course  development.  Only  a small  percentage 
of  Phase  III  resources  were  devoted  to  test  model  development. 

Although  a precise  breakdown  is  not  possible,  actual  model  development  consumed 
less  than  40%  of  resources  expended  during  Phases  I and  II. 


General  Comments  - Criticisms  and  Recommendations 

The  statements  in  this  sub-section  reflect  the  very  constructive  and  substantial 
critique  performed  by  the  T&E  team.  Once  again  the  team's  detailed  technical 
critique  was  not  included,  but  was  of  equal  quality  and  even  greater  scope. 

The  only  discontinuity  resulted  from  the  very  pragmatic  operational  orienta- 
tion of  the  T&E  team  as  it  addressed  the  more  hypothetical  areas  of  command 

and  management  philosophy  reflected  in  the  conclusions  and  recommendations  of 
Phases  I and  II. 
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SECTION  IV 
PHASE  III  PRODUCTS 


CONCLUSIONS  AND  RECOMMENDATIONS 


Based  on  the  experience  gained  through  Phase  III  and  the  conclusions  of  the 
T&E  Team,  the  following  general  conclusions  can  be  made: 

• The  Phase  III  Test  and  Evaluation  demonstrated  that  DOTS  type 
models  have  significant  potential  for  improving  the  NAVEDTRACOM 
decision  and  planning  process. 

• The  NAVEDTRACOM  Functional  Levels  will  gain  the  greatest  benefit 
from  application  of  DOTS  type  mathematical  models. 

• DOTS  type  models  can  probably  be  implemented  using  existing  person- 
nel and  without  an  increase  in  billets. 

• The  cost  of  model  application  will  probably  be  significantly  offset 
by  improved  planning  decisions.  (This  was  not  proven  during  Phase 
III  and  should  be  verified  through  additional  field  testing.) 

• It  Is  probable  that  the  three  DOTS  test  models,  modified  as 
suggested  In  the  T&E  Report's  technical  recommendations,  will  have 
utility  value.  (However,  utility  was  not  proven  during  the  T&E  and 
should  be  accomplished  through  a field  test.) 

• The  three  DOTS  models  probably  accurately  simulate  or  predict  actual 
conditions  since  they  are  based  on  the  same  planning  parameters  used 
by  current  planners.  (Phase  III  did  not  produce  an  experimental 
verification  of  model  fidelity,  and  this  should  be  considered  in  a 
more  extended  field  test.) 

t Two  negative  factors  will  Initially  retard  acceptance.  One,  training 
managers  are  not  able  to  quickly  obtain  needed  data  required  for 
meeting  their  basic  planning  and  decision  responsibilities,  and  this 
will  have  to  be  resolved  prior  to  implementation  of  sophisticated 
models.  Two,  models  could  be  Improperly  used  by  higher  level  manage- 
ment to  justify  arbitrary  resource  decisions.  Since  no  model  can 
include  all  of  the  judgmental  factors  considered  by  the  lower  level 
decision  maker,  this  improper  use  could  have  serious  Implications. 

• Proper  training  of  all  impacted  managerial  and  operational  personnel 
is  essential  to  achieving  acceptance  and  use  of  mathematical  models. 
Training  must  be  supported  by  a guided  period  of  practical  use  in 
addition  to  formal  theory  and  laboratory  sessions. 
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The  following  recommendations  are  divided  into  two  major  areas.  The  first  will 
address  possible  operational  enhancements  to  the  DOTS  test  models,  and  also 
desirable  tasks  for  enhancing  operational  use  and  acceptance.  The  second  area 
will  cover  possible  institutional  changes  that  could  lead  to  a significant  in- 
crease in  the  benefits  to  be  derived  from  mathematical  modeling. 

• Operational  Recommendations 

The  following  represents  a summary  of  the  operational  and  technical 

recommendations  identified  during  the  Test  and  Evaluation: 

1.  Initiate  data  collection  and  review  for  TRALANT/TRAPAC  applica- 
tion and  field  test  of  SCRR-TPF.  This  should  Include  a clear 
assessment  of  opportunities  for  model  contributions  at  these 
commands.  Each  command  should  designate  an  individual  from  its 
staff  who  will  be  the  primary  interface  point  in  model  develop- 
ment and  have  a clear  responsibility  for  seeing  that  the  models 
get  put  to  real  and  practical  use  with  the  command  to  the  extent 
possible. 

2.  Modify  the  SCRR  model  as  necessary  in  order  to  apply  It  at 
the  TRACOM  level. 

3.  Review  and  modify  the  TPF  model  in  terms  of  the  following  con- 
siderations: 

- Capabilities  of  a DBMS  (e.g.,  RAMIS)  to  supplant  the  TPF 

in  its  current  functions  (given  that  no  satisfactory  statis- 
tical parameters  can  be  developed). 

- Possibilities  for  more  efficient  model  design  given  that 
the  week-by-week  simulation  is  equivalent,  in  effect,  to 
successive  transformations  of  course  parameters  using  vectors 
of  proportionality  factors  on  a quarterly  basis. 

- The  possibility  for  alternative  approaches  to  the  develop- 
ment of  statistically  based  model  parameters. 

- Modifications  as  necessary  to  accommodate  TRACOM  level 
application. 

4.  Design  and  conduct  field  tests  of  the  SCRR-TPF  at  TRALANT  and 
TRAPAC. 

5.  Present  the  ETE  model  to  the  core  of  ILS  course  developers  in 
the  EDTRACOM.  This  is  to  be  followed  by  field  testing  of  the 
model  in  several  cases  of  actual  course  design.  An  assessment 
of  the  benefits  of  the  model  should  be  made  following  these 
tests  and  the  model  should  be  further  developed,  revised,  or 
put  on  the  shelf  for  operational  use  as  appropriate. 

6.  After  verification  of  fidelity  and  utility,  conduct  a cost/benefit 
analysis. 
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7.  If  operational  implementation  is  indicated  by  the  field  test, 
it  should  be  preceded  by  an  in-depth  training  program  and  a 
period  of  closely  supervised  practical  application. 
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• Institutional  Recommendations 


The  Government  T&E  was  primarily  concerned  with  the  application  of 
the  three  test  models  In  the  current  NAVEDTRACOM  decision  and  planning 
process.  In  addition,  IBM  was  tasked  with  extrapolating  the  use  of 
mathematical  models  into  a modified  NAVEDTRACOM  process.  This  Phase 
III  Report  will  not  duplicate  the  Phase  I Functional  Recommendations, 
but  will  expand  and  qualify  those  Phase  II  recommendations  having  to 
do  with  changes  to  the  NAVEDTRACOM  management  process. 

Phases  I and  II  Identified  the  following  assumptions  about  the  way 
the  NAVEDTRACOM  manages  training  and  training  resources. 

- Generally,  training  plans  and  decisions  are  formulated  by 
the  technical  staffs  and  approved  by  the  appropriate  command 
level . 

- As  long  as  these  plans  and  decisions  fall  within  acceptable 
fiscal  and  other  resource  limits,  and  are  responsive  to  user 
needs,  there  is  seldom  a challenge  from  the  authorizing  com- 
mander concerning  alternative  possibilities. 

- Under  the  conditions  described  above,  for  all  practical 
purpose,  there  is  never  a challenge  from  higher  command 
echelons  forcing  consideration  of  alternatives  by  the 
approving  commander. 

- Challenges  do  take  place  at  all  command  levels  when  there 
is  a need  to  withdraw  resources  from  a training  organiza- 
tion or  when  an  organization  requests  additional  resources. 

- Generally,  the  catalysts  precipitating  the  challenges 
described  in  the  previous  assumption  are  such  stimuli  as 
changes  in  allocations,  training  requirements,  or  other 
unanticipated  system  drivers. 

- There  is  a natural  tendency,  as  in  any  organization,  for 
any  given  level  of  command  to  limit  precise  visibility  to 
higher  levels  of  its  resource  utilization  and  Inventory. 

The  resultant  of  these  assumptions  is  a lack  of  command  pressure  to 
achieve  efficiencies  through  systematic  challenges  as  long  as  the 
NAVEDTRACOM  can  meet  training  requirements  within  a given  funding 
level.  This  also  represents  the  major  difference  between  the  way 
training  functions  are  managed  In  industry  as  compared  to  the 
military.  In  industry,  whether  supervisory  or  executive  level, 
management  is  required  to  become  very  involved  In  resource  as  well 
as  people  management.  Traditionally,  the  military  services  have 
separated  command  (getting  things  done  through  people  or  tactical 
resources)  and  resource  management  (planning  and  control  of  costs 
and  non-tactical  resources).  Most  military  commanders  at  any  level, 
in  fact,  avoid  those  tasks  associated  with  resource  management  or 
"bean  counting"  based  on  the  assumption  that  to  do  so  would  reduce 
their  effectiveness  in  achieving  primary  missions. 
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It  cannot  be  automatically  assumed  that  application  of  those  manage- 
ment techniques  used  by  industry  to  reduce  training  costs  should  be 
applied  to  the  NAVEDTRACOM.  Also,  the  assumptions  previously  stated 
should  not  be  automatically  construed  as  criticisms  of  the  NAVEDTRACOM 
management  staff.  However,  if  the  NAVEDTRACOM  is  to  achieve  the  maxi- 
mum potential  benefits  in  terms  of  cost  effectiveness  through  applica- 
tion of  mathematical  modeling,  all  levels  of  command  will  be  required 
to  adopt  their  industrial  counterpart's  techniques  of  systematic 
challenge  and  analysis  of  resource  plans  and  decisions. 

The  following  three  recommendations  are  based  on  this  assumption: 

1.  Revise  current  management  practices  and  motivational  approaches 
to  create  an  environment  amenable  to  acceptance  and  use  of  the 
recommended  management  techniques.  These  revisions  are  envisioned 
as  being  reasonable  ones,  but  will  require  changes  in  some  sensi- 
tive areas.  As  examples:  commanders  at  all  levels  will  need  to 
become  more  knowledgeable  of  the  training  planning  process  and 

its  results;  Fitness  Reports  will  need  to  emphasize  this  shift 
in  approach;  commanders  will  have  to  become  more  comfortable  with 
some  of  the  more  hypothetical  exercises  of  managerial  science 
and  operations  research;  etc. 

2.  Provide  increased  multi-level  visibility  of  training  resource 
utilization  and  inventory  to  all  appropriate  levels.  Current 
projects  within  the  NAVEDTRACOM,  under  general  direction  of  the 
CNET  Naval  Education  and  Training  Information  Service  Agency 
(NETISA),  should  eventually  provide  a basic  informational 
capability.  It  must  be  emphasized  that  this  capability  is  not 
only  essential  to  successful  Implementation  of  the  DOTS' 
institutional  recommendations,  but  also  to  those  concerning 
the  implementation  and  application  of  mathematical  modeling 
tools  in  the  current  environment. 

3.  Develop  mathematical  models  and  other  management  tools  based 

on  the  data  capability  recommended  under  the  second  recommenda- 
tion, and  designed  to  support  the  first  recommendation's 
management  practices. 

In  developing  these  recommendations,  the  distinction  between  "command" 
and  "management"  was  recognized.  It  is  understood  that  if  these  recom- 
mendations are  implemented,  commanders  will  be  required  to  practice 
certain  managerial  approaches  they  now  leave  to  their  technical  staffs. 
It  is  also  recognized  that  this  shift  may  not  be  considered  as  one  of 
"reasonable  change"  by  the  commands  impacted.  However,  the  recommended 
managerial  practices  applied  without  the  direct  force  of  command 
authority,  inspection  and  control,  will  have  limited  value  except  as 
they  assist  the  technical  staffs  to  meet  the  procedural  requirements 
of  the  Planning,  Programming  and  Budgeting  System. 

It  should  be  stressed  that  the  T&E  Team,  although  not  rejecting  the 
DOTS'  hypothesis  of  "reasonable  change,"  did  imply  that  It  would 
require  changes  too  significant  to  be  considered  reasonable.  The 
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team's  conclusion  was  not  based  on  an  assessment  of  the  DOTS'  Insti- 
tutional recommendations  as  a formal  part  of  the  T&E,  but  was  a general 
observation. 

In  support  of  the  T&E  Team's  position,  there  is  a growing  concern 
within  the  military  for  the  current  trend  towards  equating  command  to 
management.  This  is  best  illustrated  through  quotes  from  an  article^ 
by  General  Lucius  D.  Clay,  USAF: 

"Management  has  a proper  place  in  operation  of  a military 
service,  but  management  must  be  recognized  for  what  it  is  - 
a system  of  bookkeeping  that  is  primarily  associated  with 
statistics." 

"Command  is  the  relationship  between  people.  People  do 
things." 

"...let  us  not  usurp  the  traditional  functions  of  the  commander 
under  the  banner  of  management." 

General  Clay  makes  an  excellent  point.  However,  if  the  DOTS  strategic 
assumptions  pertinent  to  severe  resource  restrictions  through  the  mid- 
1 980 ' s have  validity,  military  command  may  be  forced  into  the  resource 
management  role.  If  this  happens,  mathematical  models  and  their  associ- 
ated data  bases  may  become  essential  tools  for  all  command  levels  as 
well  as  for  those  staff  levels  concerned  only  with  resource  planning. 


23Clay,  Gen.  Lucius  D.,  USAF,  Management  Is  Not  Command,  Air  Force  Maoayinp. 
Sept.  9,  1975,  Page  63. 
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PHASE  III  DOCUMENTATION 


In  addition  to  the  T&E  and  this  Phase  III  Final  Report,  which  together  constitute 
documentation  of  Phase  Ill's  R&D  outcomes,  two  additional  manuals  were  produced 
during  Phase  III. 

This  sub-section  will  provide  a brief  description  of  those  documents,  developed 
during  DOTS  Phase  III,  describing  the  three  test  models  and  providing  sufficient 
operating  procedures  to  permit  exercising  the  models  and  revalidating  their  logic 
designs.  The  documents  are  entitled  USER'S  MANUAL  and  PROGRAM  MAINTENANCE  MANUAL. 
The  Phase  III  documentation  task  was  intended  to  provide  sufficient  technical 
information  to  permit  future  testing  and  possible  model  revision  and  development 
for  inclusion  in  an  operational  environment. 

Some  limited  data  base  capabilities  were  developed  to  support  the  DOTS'  test  effort. 
These  are  also  described. 

The  DOTS  test  bed  included  installation  of  the  models  in  a commercial  time-sharing 
environment.  To  reduce  the  cost  of  establishing  a simulated  operational  test  en- 
vironment, as  well  as  reducing  the  cost  of  actual  model  development,  maximum  use 
was  made  of  existing  program  products.  Documentation  supporting  these  is  not  in- 
cluded in  the  Phase  III  documentation,  but  adequate  references  are  provided  as 
appropriate. 

A description  of  each  of  the  two  Phase  III  manuals  follows.  It  should  be  noted 
that  these  manuals  supersede  those  volumes  of  the  DOTS  Phase  II  Final  Report 
covering  model  operations  and  programming  description. 
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# PROGRAM  MAINTENANCE  MANUAL  (TAEG  REPORT  NO.  29) 

The  Program  Maintenance  Manual  provides  detailed  information  on 
the  level  of  the  three  DOTS  models  and  the  DOTS  data  base  at  the 
end  of  Phase  III.  The  level  of  information  is  sufficient  to  provide 
a base  for  programmers  intending  to  install  or  modify  the  DOTS  models. 

The  manual  is  organized  around  the  four  major  DOTS  sections,  the  DOTS 
data  base,  the  Educational  Technology  Evaluation  (ETE)  model,  the 
System  Capabilities  Retirements  and  Resources  (SCRR)  model,  and  the 
Training  Process  Flow  (TPF)  model.  The  three  models  function  inde- 
pendently and  the  data  base  (although  it  serves  both  the  SCRR  and  TPF 
models)  can  also  be  considered  as  an  independent  sub-system.  There- 
fore, each  of  the  models  and  the  data  base  are  discussed  In  a separate 
section  of  the  manual. 


Within  each  section,  the  sub-system  being  discussed  is  subdivided 
into  its  major  components.  Each  section  contains  a sub-system  de- 
scription and  macro-level  flow,  detailed  flows  and  program  listings 
of  the  sub-system  components.  For  FORTRAN  and  PL/1  programs,  the 
detailed  flow  charts  were  generated  using  the  Autoflow  II*  System. 

The  manual  is  sub-divided  as  follows: 

I .  INTRODUCTION 

A.  Purpose 

B.  Organization 

II.  DOTS  DATA  BASE 

A.  Introduction  and  Top  Level  Description 
1.  Top  Level  Flows 

B.  Data  Base  Sub-system  A 

1.  Sub-system  Top  Level  Flows 

2.  Sub-system  Program  Description 

3.  Autoflow  Diagrams,  Executive  Flows 

4.  Program  Executive  Listings 

III.  EDUCATIONAL  TECHNOLOGY  EVALUATION  MODEL 


A. 

B. 

C. 

D. 

*Autoflow  II  is 


Sub-system  Top  Level  Flows 
Sub-system  Program  Description 
Autoflow  Diagrams,  Executive  Flows 
Program  Executive  Listings 
a product  of  Applied  Research,  Inc. 
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IV.  SYSTEM  CAPABILITIES/REQUIREMENTS  AND  RESOURCES  MODEL 

(Same  as  Section  III) 

V.  TRAINING  PROCESS  FLOW  MODEL 

(Same  as  Section  III) 
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• USER'S  MANUAL  (TAEG  REPORT  NO.  30) 

The  primary  purpose  of  the  User's  Manual  is  to  provide  the  DOTS  model 
user  with  sufficient  information  to  procedurally  execute  the  models 
and  take  advantage  of  the  supporting  data  base.  In  addition,  test 
procedures  are  provided  for  each  model  to  permit  user  retesting  of 
functional  validity. 

The  manual  also  provides  an  overview  of  model  architecture  and  logic. 
Descriptions  of  input  parameters  and  outputs  are  included. 

The  manual  is  subdivided  as  follows: 

I .  INTRODUCTION 

A.  Purpose 

B.  ' Organization 

C.  System  Overview 

D.  General  System  Procedures 

II.  DOTS  DATA  BASE 

A.  Architecture 

B.  Maintenance 

1.  Administrative  Procedures 

2.  Operational  Procedures 

C.  Inquiry  Procedures 

III.  EDUCATIONAL  TECHNOLOGY  EVALUATION  MODEL 

A.  Model  Architecture 

1.  Model  Description 

2.  Assumptions  and  Logic  Design 

B.  Input  Parameter  Description 

C.  Output  Parameter  Description 

D.  Operational  Procedures 

E.  Operational  Test 

IV.  SYSTEM  CAPABILITIES/REQUIREMENTS  AND  RESOURCES  MODEL 

(Same  as  Section  III) 

V.  TRAINING  PROCESS  FLOW  MODEL 

(Same  as  Section  III) 
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